New Paper: Towards a science of AI agent reliability

There's a strange gap in AI right now that nobody's been able to name properly, and a new paper from Princeton researchers Stephan Rabanser, Sayash Kapoor, and Arvind Narayanan finally puts a label on it. Everyone keeps pointing at benchmark scores climbing month over month, then wondering aloud why AI agents still aren't running warehouses or handling customer service without a human hovering nearby. The answer, it turns out, isn't capability. It's reliability. And the industry has been measuring the wrong thing entirely.

The team borrowed a page from aviation and nuclear safety engineering, fields that figured out decades ago that average performance and reliability are not the same animal. A reactor that shuts down correctly 99% of the time but fails unpredictably the other 1% isn't safe, it's a lawsuit waiting to happen. So the researchers broke reliability into four core dimensions: consistency (does the agent get the same answer twice), robustness (does it hold up when instructions are phrased slightly differently), predictability (does it know when it's wrong), and safety (when it fails, is the failure survivable). They expanded that into 12 measurable metrics and ran 14 models from OpenAI, Google, and Anthropic through 500 total benchmark runs across two testbeds, GAIA and TauBench.

The results are almost uncomfortable to read if you've been riding the AI hype wave. Accuracy shot up substantially across 18 months of releases. Reliability barely budged. Consistency scores ranged from just 30% to 75%, meaning a model might nail a task once and flub it on the next identical attempt. Calibration, meaning whether an agent's confidence actually maps to correctness, was the weakest link across the board; on one benchmark, most models couldn't tell their right answers from their wrong ones any better than a coin flip. Rephrasing instructions with identical meaning tanked performance too, even though the models handled genuine technical glitches like server timeouts just fine. And bigger models weren't automatically better here. Scaling helped calibration and robustness some, but it actually made consistency worse in several cases, likely because larger models have more behavioral range to wobble within.

The paper's authors are careful to hedge their own conclusions, which is refreshingly honest for a field that loves bold claims. Maybe accuracy at 99% is good enough regardless of predictability, they concede, though they push back on that themselves, arguing that high-stakes autonomous deployment realistically needs three to five nines of reliability, not two. Maybe reliability gains just look slow relative to eye-popping accuracy gains, without being slow in absolute terms. They test that too, and note that a naive linear projection would put agents at 100% reliability in three years, which they rightly call absurd, since each additional nine of reliability tends to get exponentially harder, not linearly easier.

What's most useful here isn't the diagnosis so much as the prescription. The researchers argue that benchmarks need to stop reporting a single accuracy number from one run and start reporting reliability profiles: multiple runs, paraphrased instructions, injected faults, tracked over time. They're launching a reliability index to push the industry toward that standard. Whether developers actually adopt it, or whether reliability just quietly becomes the excuse everyone reaches for when their agent-powered product underperforms in production, remains the open question.