Most coding agent benchmarks skip large-scale refactoring. Not this one.
The New Stack Meredith Shubel
AI coding agents face a new refactoring test and the best model only solved 41.2%. It’s a reminder that flashy benchmark scores can hide how badly these tools handle real code changes.
Based on reporting by The New Stack, Meredith Shubel — read the original for the full story.
Summary, retelling and take written by AI under human oversight; images are AI-generated illustrations. How we work · Report an error
AI coding agents still look shaky when the job is not “write a snippet,” but “change a real codebase without breaking it.” A new benchmark built by researchers at Shanghai Jiao Tong University, Peking University, Douyin Group, and others puts that weakness on display: the best model only reached a 41.2% resolve rate.
The benchmark, SWE-Bench ProMax, was built around 170 refactoring cases pulled from real commits in seven languages: Python, Java, TypeScript, Go, C, C++, and Rust. Its creators did more than collect tasks. They rewrote issue descriptions to be clearer, manually checked test suites to weed out tests that were too narrow or too broad, and dropped problems that were too small or didn’t reach across enough files.
That curation matters because AI benchmark results are easy to flatter and hard to trust. The researchers point to a recent audit showing that nearly 60% of unsolved SWE-bench Verified instances had flawed tests. And once a benchmark gets popular enough, models may have already seen the answers in training, which makes the score feel cleaner than the underlying capability really is.
The deeper issue is that large-scale refactoring asks for more than pattern matching. Shane Warden of ActiveState argues that strict refactoring has zero tolerance for error, behavior changes, or broken reversibility, and he doesn’t buy the idea that token proximity equals structural understanding. Vojtěch Pavlík of SUSE says the same problem shows up when code meets time: race conditions, retry bugs, and failures that only appear when things happen in the wrong order.
SWE-Bench ProMax is basically a warning shot. Most agent benchmarks are built to be quick and deterministic, which pushes them toward small, tidy tasks and away from the ugly, cross-file work that real engineers actually fear. If a model can ace the easy stuff and still stumble at refactoring, the scorecard has been telling a better story than the product deserved.
My take — AI-written commentary, not fact-checked reporting
Benchmarks love the neat little problems because neat little problems are easy to grade. That’s fine for a leaderboard, and useless for a codebase with history, dependencies, and human mistakes baked in. The more interesting move now is not bigger hype, but better tooling that forces models to prove they can respect structure instead of just spraying diff-shaped confetti.
Read more about this at: The New Stack