Evaluate any agent framework with Amazon Bedrock AgentCore Evaluations
Amazon Web Services Swarnim Singhal
AWS says AgentCore Evaluations can score agents built with different frameworks. It works by reading OpenTelemetry traces, so you don’t have to rebuild evals for each SDK.
Based on reporting by Amazon Web Services, Swarnim Singhal — read the original for the full story.
Summary, retelling and take written by AI under human oversight; images are AI-generated illustrations. How we work · Report an error
AWS is trying to fix a problem that gets uglier as agent frameworks multiply: evaluation tools usually expect one very specific stack, and production teams rarely stay that tidy. If you built in LangGraph, LlamaIndex, the OpenAI Agents SDK, Google ADK, the Claude Agent SDK, or Strands Agents, you can still end up on the same AgentCore runtime — but until now, the evaluation layer was often the brittle part.
AgentCore Evaluations takes a different route. Instead of caring which SDK produced the agent, it looks at OpenTelemetry data flowing through CloudWatch and rebuilds the session from spans and event records. The key pieces are simple: one session is grouped by session.id, one trace is one user turn, and the evaluators only need three roles to do their job — the top-level invoke agent span, the inference spans for model calls, and the execute tool spans for tool use.
Everything else is treated as context. Retrieval, reranking, guardrails, memory reads and writes, orchestration steps — those can all be present, and the service will skip over what it doesn’t need. That matters because real traces are messy, and the point here is that messiness no longer becomes a setup problem. The system understands both OpenTelemetry GenAI and OpenInference conventions, and it can follow either schema to the same result.
The framework support story is broader than the named list, too. AWS says any library that uses a scope name under opentelemetry.instrumentation.* or openinference.instrumentation.* can be read through a generic path. That makes the scope-name prefix the opt-in signal. If a team invents its own custom scope name, even with perfect spans, it won’t be picked up.
There are still a few practical catches. The agent’s spans need a session.id that matches the runtimeSessionId, and the data source has to include message content, not just spans. On newer agents with unified observability, that’s straightforward. On older setups, spans can live in aws/spans while message content sits separately, and if the source only points at the spans, response-quality evaluators will fail because the content is missing.
The actual scoring layer is the same regardless of framework. AWS says GoalSuccessRate, Correctness, Helpfulness, and custom LLM-as-a-judge evaluators all work on the reconstructed session. The pitch is not glamorous, but it is the kind of plumbing teams quietly beg for: one eval path, many frameworks, less ritual sacrifice to instrumentation drift.
My take — AI-written commentary, not fact-checked reporting
This is the right kind of boring. Everyone loves a new agent framework; nobody enjoys discovering that evaluation only works if the traces were blessed by a very specific SDK. The real product here is less AI magic than standards discipline, which is exactly why it matters.
Read more about this at: Amazon Web Services