SOP-Bench: A new benchmark for evaluating AI agents on real business procedures

Amazon Science has put out SOP-Bench, a benchmark built to see whether AI agents can actually carry out real standard operating procedures instead of just sounding smart about them. The target is the kind of work organizations run on every day: patient intake, dangerous-goods checks, customer service, content moderation, financial compliance, warehouse inspection. The twist is that the procedures are left as humans wrote them, with the gaps, ambiguities, and assumptions intact.

That matters because SOPs are exactly where polished demos tend to crack. A procedure may say “verify insurance” twice, but a seasoned employee knows those are two different checks. A model does not get that background for free. It has to infer the right meaning, keep track of earlier steps, and choose between tools that may look similar but do very different things.

SOP-Bench is meant to close a gap in existing agent testing. Amazon says it is the first benchmark to combine genuine enterprise procedures, working tools, and ground-truth answers so an agent is scored on completing the task, not on producing text that flatters an automated judge. The release includes more than 2,000 tasks across 12 business areas, along with tool interfaces and correct outcomes. It also ships as a framework, so teams can plug in their own agents and even add their own procedures.

The benchmark was presented at the 2026 Conference on Knowledge Discovery and Data Mining, and Amazon says it was built with domain experts and Anthropic Claude 3.5 Sonnet v2 doing the mechanical conversion work. The experts wrote the original procedures, checked the logic, verified the data, and reviewed the generated code. Amazon says no proprietary or sensitive data was used.

The early results are not kind to the usual “just upgrade the model” instinct. On one reasoning-style setup, Claude 4.5 scored lower than Claude 4. Adding tools can also backfire: in one video-annotation procedure, performance nearly halved when the required six tools were buried inside a larger set of 26. And there was no universal winner. Some tasks were easy, like triaging emails by intent, where agents were right about nine times out of ten. Others, like annotating objects in a driving video, were right only about one time in four.