How to Evaluate AI Agents: From Tool Calls to Task Completion

The big shift in agent evals is simple: stop asking whether the model sounds right and start asking whether the job actually got finished. A single answer string was fine for old-school LLM tests. It falls apart when the system has to make dozens of tool calls, handle failures, and keep state across a live environment.

That is why tool use has become the connective tissue of serious agent benchmarks. BFCL helped by checking function choice and argument quality across single- and multi-turn setups, but it still only scores isolated calls. A correct refund call can still be a useless one if the agent skipped the checks and updates that make the refund real.

The deeper eval stack now has two layers. Step-level scoring looks at whether each call was valid and useful in context. End-to-end scoring ignores the path and checks only the final state: did the ticket route, did the refund post, did the database change, did the issue get fixed? That split matters because debugging needs the trace, while users only care about the outcome. The trace is the log of the attempt: user message, steps taken, and the environment state at the end.

Once you start reading benchmarks this way, a lot of familiar metrics snap into place. Success rate is the release gate. Consistency across 3–5 trials matters because 90% in one run and 74% in another is not the same thing as 84%. Tool-call precision catches hallucinated tools and extra calls. Argument accuracy catches the case where the right API gets the wrong payload. Step count and cost matter too, because a model that gets there in fewer steps and less spend is doing better work, not just prettier work.

The source also makes a blunt point about comparability: two benchmarks can both claim to test tool calling and still mean very different things. A single-turn benchmark with one tool does not tell you much about planning, state drift, or error recovery. Executable verification is the cleanest test when it exists. Judges can fill gaps, but only provisionally, and contamination is now a live problem when web-searching agents can scrape answers on the fly. In that world, private domain evals are attractive for a very boring reason: they can’t be scraped.

That’s the practical angle NVIDIA is leaning on with Nemotron 3.5 Lightning. Its public scores are framed around task completion and time-to-done, not just isolated call accuracy. On PinchBench, it reaches 86% accuracy while finishing 10,000 tasks 30% faster than Qwen3.6 35B at comparable accuracy. That is the kind of comparison that actually maps to deployment: the model that finishes the work cleanly beats the one that merely looks clever in a benchmark table.