New paper: Emerging trends in AI agent evaluation
AI as Normal Technology Sayash Kapoor ● Covered by 2 sources
New paper says AI agent evals are getting messy, costly, and harder to trust. The big issue isn’t just scores anymore; it’s whether agents actually did what the test says.
Based on reporting by AI as Normal Technology, Sayash Kapoor — read the original for the full story.
Summary, retelling and take written by AI under human oversight; images are AI-generated illustrations. How we work · Report an error
A Princeton-led group has put out a new paper on a problem that is getting bigger as AI agents spread: how to judge them without fooling yourself. The authors, working with people from Stanford, the UK AI Security Institute, and Brookings, say the field needs better answers on what counts as a good evaluation, what should be evaluated, and who gets to do the evaluating.
Their central point is simple enough. For agents, looking only at the final result is not enough. Evaluators have to inspect the whole trail, because agents can use tools, take shortcuts, and behave differently depending on which component changes. That makes the work slower, more technical, and more expensive. The paper says some agents on the ExploitBench benchmark cost tens of thousands of dollars in credits just to evaluate, and that still leaves out the engineering needed to build and maintain the setup.
Then there is the scale problem. Agents generate huge numbers of traces, and the tools used to monitor them are imperfect and need checking too. In one example the authors cite, evaluators looking back at the OpenAI-HuggingFace incident used other agents to search for harmful actions, while also acknowledging those tools had biases and limits of their own. Add dynamic environments and agents working alongside other agents, and even the metrics get slippery.
The paper also argues that evaluation needs to widen out. A system can look strong in the lab and still disappoint in a specific use case. METR’s randomized controlled trial, for example, found that use of agent tools for software development increased overall completion time on the tasks studied. So the authors want more work across three layers: what agents can do, how they interact with humans, and what they do to institutions and incentives.
Who gets to evaluate matters too. The paper says meaningful independence depends on funding, access to models, access to Chain of Thought traces, and freedom from retaliation. And it makes a practical pitch to evaluators, labs, funders, conference organizers, leaderboard operators, deployers, and policymakers: treat agent evaluation as its own discipline, with higher standards, broader participation, and more openness about the costs.
My take — AI-written commentary, not fact-checked reporting
This is what happens when a field mistakes leaderboard theater for measurement. Agent evals are becoming a real profession, not a side quest, and the people with the least access are often the ones closest to the truth. That should make labs nervous, which is usually a sign the system is working better than the marketing department is.
Read more about this at: AI as Normal Technology