Evaluating Agents on Highly Subjective Tasks
Deep (Learning) Focus
AI agent evals are messy, especially for open-ended tasks. That’s why the piece pushes human review, stats, and monitoring instead of trusting one score.
Based on reporting by Deep (Learning) Focus — read the original for the full story.
Summary, retelling and take written by AI under human oversight; images are AI-generated illustrations. How we work · Report an error
Evaluating AI agents is still one of the ugliest parts of the job. The field keeps shipping new benchmarks, then arguing about whether they’re saturated, benchmark-maxxed, or simply bad at measuring what agents actually do. This overview, written for an AI Engineer conference talk, takes the boring but necessary route: if evaluation is messy, build it like a system, not a scorecard.
The core problem is simple enough. Agents are non-deterministic, they work across a wide range of tasks, and many of those tasks are long-horizon and expensive to test. So a single benchmark gain can be misleading. It may reflect lucky runs, a narrow task mix, or a grader that doesn’t match how the agent is used in practice. And once the task gets subjective — creative writing is the obvious example — the clean pass-fail world falls apart.
The piece argues for a three-part roadmap: human evaluation, offline evaluation, and production monitoring. That’s the practical spine of the whole thing. The idea is to combine it with statistical tools that help separate real improvement from noise, plus data curation methods that make evaluation sets more diverse, realistic, and informative. In other words, fewer random tasks, more useful ones.
A lot of the discussion is about how to handle verification. Some tasks can be checked deterministically with test cases, string matching, or symbolic verification. Others need human judges or LLM judges, and those bring their own bias and disagreement. Rubrics help by breaking quality into specific criteria instead of one fuzzy overall score, but the judges still need to be calibrated against a golden dataset of human examples.
The statistics section is the part most teams skip, which is exactly why they shouldn’t. Power analysis helps estimate how many samples are needed before running an eval. Confidence intervals put uncertainty around the result. Re-sampling helps tame non-determinism by running the same task more than once. None of that makes evaluation glamorous. It does make it less wrong.
My take — AI-written commentary, not fact-checked reporting
The field keeps pretending one benchmark score will save it, which is adorable in a sad sort of way. For agents, especially subjective ones, the real product is not a leaderboard number but a measurement pipeline that can survive noise, judges, and reality. That’s less sexy than a shiny chart, and exactly why it matters.
Read more about this at: Deep (Learning) Focus