The Most Common Questions About Agent Evals, Answered

If you’re building an agent, testing it like ordinary software won’t get you far. The same request can produce different outputs, more than one answer may be acceptable, and an agent can land on the right result while taking a very odd route to get there. It might call a tool it didn’t need. It might also take an action the user never approved.

That’s why the article pushes agent evals as a separate discipline. The starting point is traces: records of a run that include inputs, outputs, model calls, and tool calls in between. Read a handful, pick one behavior you care about, and build a small dataset around it. Ten to twenty examples is enough to begin, and failures from production or testing should keep feeding that dataset back into the loop.

The split between offline and online evals matters. Offline evals run before a change ships, against a fixed dataset with reference outputs when you have them. Online evals score live production traces, where there often isn’t a reference answer, so they rely on code checks and LLM judges. The two are meant to reinforce each other: online failures become new offline examples, and offline runs catch regressions before users see them.

The article also draws a clear line between evaluator types. Code evaluators are deterministic and cheap, but limited. LLM-as-a-judge can handle messy judgments like whether a reply clearly confirms flight details. Human review is slower but remains the best check for subjective quality. Pairwise evaluators are handy when the choice is between two prompts or models. And any of these can score a single step, a final response, a whole conversation, or even the end state of a mocked environment.

Benchmarks are treated as a more structured kind of offline eval. They use isolated environments in containers and score what the agent leaves behind, not just what it says. The article points to Harbor as the open source harness behind Terminal-Bench and says it can log runs into LangSmith experiments. For production traffic, the advice is simple: sample, filter, use code checks where possible, and reach for a cheaper judge when you can. Not every trace deserves a premium model's opinion.