AI’s best coding agent fails 60% of the time — and the data backs it up
The New Stack Amanda Caswell ● Covered by 2 sources
Claude Fable 5.1 topped a new coding test with 38.8% — and still failed most of the time. It was tested on private company code, so the misses look a lot less theoretical.
Based on reporting by The New Stack, Amanda Caswell — read the original for the full story.
Summary, retelling and take written by AI under human oversight; images are AI-generated illustrations. How we work · Report an error
Claude Fable 5.1 won a new coding benchmark, but the headline number is a blunt reminder of how brittle these systems still are. On Real-SWE, it scored 38.8% and failed more than six out of 10 attempts. That is the best result in the set, not a flattering one.
Real-SWE comes from Y Combinator-backed Specific Labs, and it tests agents on private codebases from real companies rather than public repository problems. The goal is obvious: make the tasks look more like actual engineering work, where the code is unfamiliar and the answers are not sitting in the training data. Specific Labs says it can’t guarantee a model has never seen any of the code, but it argues private code makes that far less likely. It also says 99% of tokens in real-world enterprises are hidden from frontier models.
The benchmark also bakes in the full coding stack, not just the model. Fable 5.1 ran through Claude Code, GPT-6 Astra through Codex CLI, and Gemini 3.8 Flash through Gemini CLI. That matters because tooling changes outcomes. The same model can look very different depending on the scaffolding around it.
The spread in scores was wide. Astra came next at 33.8%, Gemini 3.8 Flash scored 31.2%, GLM 5.3 reached 28.8%, Grok 4.6 and Muse Spark 1.3 tied at 23.8%, Kimi K3 got 18.8%, and GPT-5.6 Sol ended at 16.2%. Each model got eight tries per task. On the hardest problems, the floor dropped even lower: six of the 10 tasks landed below 15%, and one tax jurisdiction bug was fixed only 3.1% of the time.
Real-SWE’s tasks also cut across more of the codebase than many benchmarks do. Its solutions touched a median of 11 files, compared with a six-file median in FrontierCode and DeepSWE. The benchmark drew from proprietary code licensed from real businesses, including a consumer product with more than 200,000 users and a fintech platform that has processed more than 100,000 bank statements. And yet no model solved the analytics stream reducer across 64 attempts. That is the real story here: even the best agent in the lineup was still mostly getting lost in unfamiliar production code.
My take — AI-written commentary, not fact-checked reporting
This is the part the industry keeps trying to talk around: a model can top a benchmark and still look shaky in the kind of codebase people actually get paid to maintain. Private-code tests are useful because they remove some of the training-data confetti from the room. The result is less hype, more reality, and a lot more failure than the demos suggest.
Read more about this at: The New Stack