Monitoring production agent lifecycle with AWS DevOps Agent and AgentCore Evaluations
Amazon Web Services Meghana Ashok
AWS says some agent failures look fine in the logs but still break the user experience. Its new setup watches both quality and infrastructure, because green metrics can still mean a bad answer.
Based on reporting by Amazon Web Services, Meghana Ashok — read the original for the full story.
Summary, retelling and take written by AI under human oversight; images are AI-generated illustrations. How we work · Report an error
AWS is pitching a simple idea with messy consequences: if you only watch the infrastructure, you’ll miss what production agents are actually doing. An agent can fail to call its foundation model and return nothing because of a missing IAM permission. Or it can keep humming along, hit every tool, and still steer a user the wrong way.
That gap is why AWS built a production airline reservation system around two monitoring layers. One layer is Amazon Bedrock AgentCore Evaluations, which continuously scores live interactions for things like helpfulness, correctness, and goal completion. The other is AWS DevOps Agent, which investigates infrastructure problems by tracing logs, permissions, and service calls without waiting for a person to piece it together by hand.
The demo system uses four specialized agents in a Swarm pattern: a Supervisor Agent, a Flight Agent, a User Agent, and a Reservation Agent. The setup has to handle awkward, real-world requests like multi-city trips, loyalty certificates, and company travel policy in one conversation. That’s exactly where agent systems get brittle. A request can move through several handoffs, and there isn’t a fixed call graph to instrument. Failures can show up at different points each time.
AWS says AgentCore Evaluations samples a configurable slice of production traffic, then scores interactions in the background with reasoning attached to each score. When quality drops, it looks for patterns across low-scoring sessions and suggests changes to prompts, tool selection, or orchestration. AWS DevOps Agent, meanwhile, pulls CloudWatch logs, builds a topology graph, correlates errors across IAM, Amazon Bedrock, and the agent runtime, and returns root-cause analysis plus remediation advice.
The architecture ties all of this together through a React frontend on AWS Amplify, Amazon S3 session storage, AgentCore Identity, and AgentCore runtime. Monitoring data from the runtime goes into Amazon CloudWatch in OpenTelemetry format, and evaluation results land there too. The source code, CDK infrastructure, evaluation dashboard, and DevOps Agent integration are available in GitHub, along with references to FAST and the AgentCore Evaluations guide.
My take — AI-written commentary, not fact-checked reporting
This is the right call: stop pretending agent ops is just server ops with nicer branding. A green dashboard means very little if the system is quietly booking the wrong thing, and the industry keeps learning that lesson the expensive way. The mildly funny part is that the fix is basically two old ideas—measure the user-facing output and investigate the plumbing—finally dragged into the same room.
Read more about this at: Amazon Web Services