The Sequence AI of the Week #895: OpenAI's Show Us Where Coding Evals Break
Substack Jesus Rodriguez
OpenAI audited SWE-Bench Pro and found nearly a third of the tasks are broken. That precise 80.3% score everyone's citing is measuring the test's flaws as much as the model's skill.
Based on reporting by Substack, Jesus Rodriguez — read the original for the full story.
Summary, retelling and take written by AI under human oversight; images are AI-generated illustrations. How we work · Report an error
There's a comforting illusion in benchmark scores: a number like 80.3 percent feels rigorous, almost scientific. OpenAI's latest audit of SWE-Bench Pro punctures that illusion pretty thoroughly. Precision, it turns out, is not the same thing as validity.
SWE-Bench Pro was built to fix problems with earlier coding evals — shorter tasks, contaminated training data, unrealistic repos. On its 731-task public split, frontier model scores climbed from 23.3 percent to 80.3 percent in just eight months. That's a huge jump, the kind of curve that gets screenshotted into pitch decks. OpenAI decided to ask the less comfortable question: how much of that gain is real capability, and how much is the benchmark itself giving ground it shouldn't?
The numbers are not flattering. OpenAI's own agent-assisted audit flagged 200 of the 731 tasks, 27.4 percent, as defective — tasks that reject valid solutions, accept incomplete ones, or demand behavior the prompt never actually asked for. A separate review using experienced human engineers was even harsher, marking 249 tasks, 34.1 percent, as broken. Two independent methods, roughly a third of the benchmark unreliable either way.
OpenAI has since pulled back its earlier endorsement of SWE-Bench Pro as a benchmark worth adopting. The deeper point isn't really about this one eval, though. It's that coding benchmarks work as executable specifications only when the specification itself is trustworthy, and apparently nobody was checking that closely enough until agents were pointed at the grading logic rather than the code.
There's an odd irony in the fix: the same coding agents whose abilities we're trying to measure turned out to be useful for auditing the measurement tool. That's not a small detail — it suggests the near-term role for agentic AI in benchmarking isn't just taking tests, it's grading the graders.
My take — AI-written commentary, not fact-checked reporting
I've said for a while that leaderboard chasing is the industry's favorite distraction, and this is exhibit A: a third of a widely-cited benchmark was quietly broken while everyone treated the decimal point as gospel. If a lab as resourced as OpenAI only found this after building an agent to check it, imagine what's rotting in the dozens of smaller, less-scrutinized evals people cite in marketing slides every week.
Read more about this at: Substack
