Monitor Amazon SageMaker Pipelines cross-account with custom Amazon CloudWatch dashboards
AWS Giorgio Pessot
AWS shared a way to watch SageMaker ML pipelines running across multiple accounts and regions from one screen. Instead of hopping between consoles, teams get a single live dashboard for everything.
Based on reporting by AWS, Giorgio Pessot — read the original for the full story.
Summary, retelling and take written by AI under human oversight; images are AI-generated illustrations. How we work · Report an error
Keeping tabs on machine learning pipelines gets messy fast once a company spreads its workloads across several AWS accounts and regions. That's the problem AWS is tackling with a new reference solution built around Amazon SageMaker Pipelines and Amazon CloudWatch, aimed at engineers tired of logging into one console after another just to check whether a training job finished.
The setup uses a hub-and-spoke design. A primary account hosts the actual dashboard, DynamoDB tables, and the Lambda functions that crunch the data, all bundled into what AWS calls the Dashboard stack. Every other account or region being watched runs a much lighter Forwarder stack, which listens for SageMaker events, adds context like execution status or display names, and pushes everything over to the hub through Amazon EventBridge. IAM roles and resource policies handle the cross-account security side of that handoff.
Once the data lands centrally, it flows into DynamoDB and gets rendered through custom Lambda-powered widgets inside a CloudWatch dashboard called PipelineMonitoringDashboard. From there, someone can see account IDs, regions, execution times and statuses at a glance, filter by pipeline name, or drill into individual step details, things like start and end times or step type. There's also an alarm layer: if calls to the dashboard's backend Lambda functions spike past defined thresholds, CloudWatch fires an alert through an SNS topic that's encrypted with a customer-managed KMS key.
AWS built the whole thing to be event-driven and serverless rather than relying on constant polling, which keeps costs and maintenance lighter than an always-on monitoring stack would. The company also published a GitHub repository with a full AWS CDK example, plus deployment and cleanup instructions, so teams can spin it up, test it against real pipeline executions, and tear it down again when done. A few requirements apply, including specific versions of Python, the CDK, and the AWS CLI, along with Docker for packaging the Lambda functions.
AWS frames this as a flexible starting point rather than a finished product. The post suggests extending it to cover Step Functions, AWS Batch or Glue jobs, or EMR clusters, and mentions options like deploying inside a VPC for tighter security, wiring in CI/CD pipelines through AWS Organizations and the AWS Deployment Framework, or swapping in Amazon Managed Grafana for visualization instead of sticking with CloudWatch alone.
My take — AI-written commentary, not fact-checked reporting
This is a sensible fix for a problem that shouldn't exist in the first place: nobody should have to babysit five different console tabs to know if a pipeline failed. The serverless, event-driven approach is the right instinct, cheaper than always-on polling and it scales without extra babysitting. What's notable is that AWS is handing this out as a DIY CDK template rather than a managed product, which says something about how niche cross-account MLOps observability still is, even as multi-account setups become the default for any org running ML at scale.
Read more about this at: AWS