Introducing SWE-bench Verified
OpenAI
OpenAI cleaned up SWE-bench, the coding test everyone uses to brag about AI models. The old version was full of broken or unfair test cases skewing scores.
Based on reporting by OpenAI — read the original for the full story.
Summary, retelling and take written by AI under human oversight; images are AI-generated illustrations. How we work · Report an error
Benchmarks are only as good as the mess underneath them, and SWE-bench had a lot of mess. The test, which asks AI models to fix real GitHub issues pulled from popular Python repositories, became the go-to yardstick for coding agents over the past year. Everyone from Anthropic to Cognition to random open-source labs started quoting SWE-bench numbers like they were gospel. Turns out a decent chunk of those numbers were built on shaky ground.
OpenAI dug into the original dataset and found problems that anyone who's actually written unit tests would recognize instantly: test cases so strict they'd reject a technically correct fix, issue descriptions missing the context needed to solve them, and setups where the evaluation environment itself was flaky. None of this is scandalous — building a benchmark from scraped GitHub issues is messy work — but it does mean prior scores were noisier than people assumed.
So OpenAI paid human annotators, mostly experienced software engineers, to go through the benchmark by hand and flag what should stay. The result is SWE-bench Verified, a 500-task subset that's been checked for solvability and fairness. It's smaller than the original by design — better to have 500 tasks you trust than 2,000 you have to squint at.
The practical effect is that model comparisons on this benchmark should now mean something closer to what people assume they mean. OpenAI ran its own models against the new set and, unsurprisingly, saw scores shift compared to the old benchmark, in some cases significantly. That alone tells you how much noise was baked into the original numbers being cited in papers and marketing decks all year.
This kind of housekeeping doesn't generate headlines the way a new model release does, but it matters more than people give it credit for. Coding ability is becoming one of the main battlegrounds for frontier models, and if the ruler you're using to measure progress is bent, every claim built on top of it is suspect too.
My take — AI-written commentary, not fact-checked reporting
I'll say the quiet part: half the 'SOTA on SWE-bench' claims from the last year were probably measuring benchmark noise as much as model skill, and nobody wanted to admit it because the number was convenient for a launch tweet. Cleaning up eval data is unglamorous work compared to shipping a new model, which is exactly why it took this long and why more benchmarks badly need the same treatment.
Read more about this at: OpenAI
