Evaluate skill-equipped agents with Strands Evals and Amazon Bedrock AgentCore
Amazon Web Services Sangmin Woo
AWS added tools to check if AI agents pick the right skill and actually follow it. That’s the missing part when a fluent answer still skips the business rules.
Based on reporting by Amazon Web Services, Sangmin Woo — read the original for the full story.
Summary, retelling and take written by AI under human oversight; images are AI-generated illustrations. How we work · Report an error
AWS is leaning into a very practical problem with agentic AI: a good answer can still be the wrong one. The company’s new pitch is that when agents use modular skills — reusable instruction sets often stored in a SKILL.md file — you need to evaluate both whether the right skill was chosen and whether it was actually followed.
That matters because skills are supposed to carry the messy parts of real work: compliance checks, document workflows, escalation rules, team conventions. If an agent routes to the wrong playbook, or only half-uses the right one, a final answer can still look polished enough to fool a simple quality metric. AWS says that’s exactly why the final response alone is not enough.
To catch that, Strands Evals SDK and Amazon Bedrock AgentCore Evaluations add three checks. Skill Selection Accuracy asks whether the invoked skill fit the task. Skill Instruction Following grades how fully the agent followed the skill’s steps. Skill Invoked, available in Strands Evals, is a deterministic check that a named skill was actually loaded. AWS says the judge-based evaluators return per-invoked-skill results, which keeps multi-skill runs debuggable instead of turning them into one mushy score.
The company’s example is an HR assistant with separate PTO and benefits skills. If someone asks about dental and vision benefits but the agent reaches for the PTO skill, the response may still sound fine. The selection evaluator isolates that bad routing. If the agent gets the right PTO skill but skips rollover rules, the instruction-following evaluator catches the missed step. Same answer surface, different bug.
AWS also shows how to use the checks in practice: run them in Strands Evals during development or continuous integration, or point AgentCore Evaluations at OpenTelemetry traces in production. There’s support for on-demand, batch, and continuous online evaluation, and the company says continuous sampling is useful for catching catalog drift, unanticipated phrasing, and long-session failures. The broader message is simple: if agents are going to be built from skills, the tests have to be skill-shaped too.
My take — AI-written commentary, not fact-checked reporting
This is the right kind of unglamorous AI tooling. Most teams don’t need another bigger model; they need a way to prove the agent used the right playbook and didn’t freestyle through the compliance section like it was optional reading. Open standards for skills are doing more useful work here than another round of prompt cosplay.
Read more about this at: Amazon Web Services