AI agent evaluations are part of the product

A polished demo can hide a lot. A team tests an agent with a few realistic questions, gets the answers it wants, records the run, and ships. Then the retrieval setup changes, a model gets upgraded, and the same agent starts missing a required citation on one task and picking the wrong customer lookup tool on another. The bug may only show up when users complain or monitoring catches it. That's the core warning here: a one-off success doesn't prove the next release will behave the same way.

The fix starts before the tests do. Teams need to write down what the agent is supposed to do, what it must not do, and what counts as a failure. For a support agent, that could mean answering billing questions from the right account, citing the current policy, refusing to expose another customer’s data, and asking for an account number when needed. The article pushes a simple but useful distinction: the right answer is not enough if the agent got there by retrieving the wrong document, calling an unnecessary tool, or escalating a routine request it should have handled itself.

That means evaluating the whole path, not just the final line. The trace should include the request, system instructions, model and app build, prompt and retrieval configuration, tool schemas, retrieved sources, tool calls, permissions checks, and the final response. It should also capture latency, token use, and cost. Some things can be scored by a human or judge model, like whether the answer actually addressed the request. Others are hard gates: cross-tenant retrieval, missing approval, or an unapproved update should fail no matter how nice the prose looks.

The source makes a good case for keeping the first test set small and real. Ten tasks from support tickets, workflow logs, incident reports, and user conversations beat a giant benchmark full of fake prompts. Scenarios should include missing data, outdated documents, tool timeouts, multi-turn workflows, and weird-but-valid requests. And each case needs frozen fixtures so the team can reproduce the run instead of arguing about what the model might have seen. A failure you can’t replay is just a story.

It also helps to separate fixed rules from variable scores. Task completion, factual support, policy compliance, user experience, latency, and cost are not the same thing, and they shouldn’t be crushed into one mushy number. Compare each new build against a baseline on the same scenarios. If a version is slower but clearly better, that’s a product call. If it bypasses one permission gate, it’s out. That’s the whole point of turning evaluation into a release gate instead of a postmortem ritual.