Not all model upgrades are upgrades
Microsoft ● Covered by 2 sources
A new Claude model looked cheaper on paper but used up to 12x more tokens on some tasks, sometimes making it pricier overall. Testers found quality even dropped on architecture work, proving upgrades aren't automatically better.
Based on reporting by Microsoft — read the original for the full story.
Summary, retelling and take written by AI under human oversight; images are AI-generated illustrations. How we work · Report an error
Everyone assumes a newer, cheaper-per-token model is a free upgrade. A team testing Claude Sonnet 4.6 against Claude Sonnet 5 inside GitHub Copilot Chat found that assumption doesn't hold up once you actually measure what happens.
Sonnet 5 cut per-token pricing by 33% across input, cached input, and output. That looks like a clean win until you factor in consumption. Across 150 runs spanning 15 scenarios, Sonnet 5 burned through roughly 12x more tokens at the median on architecture tasks, with one run hitting 47x the typical volume. On SharePoint Framework code upgrade tasks, the gap was around 10x. A 33% discount doesn't survive that kind of multiplier. On code upgrades, Sonnet 5 ended up costing $2.01 per run versus $0.55 for the older model, nearly four times as expensive. Architecture work flipped the script: Sonnet 5 averaged $0.47 per run against $0.54, a modest win, but only because the token increase there was milder.
Cost aside, quality didn't reliably improve either. Both models hit the same 75% completion rate on the Select gate for architecture tasks, but Sonnet 4.6 scored 90% on the Idiomatic quality measure compared to Sonnet 5's 78%, and the older model matched or beat the newer one in 8 of 9 comparable scenarios. On an IoT analytics design task, Sonnet 4.6 passed idiomatic checks in 4 out of 5 runs; Sonnet 5 managed just one, on the identical prompt.
Code upgrades told a different story. Sonnet 4.6 passed the Select gate only 60% of the time, while Sonnet 5 hit 100%, and it was far more reliable at following exact version instructions, correctly using SPFx v1.22.0 every time while the older model kept substituting v1.22.1 from documentation. One Sonnet 5 run even consumed 69 million tokens digging up undocumented migration steps, though that depth showed up in only one of five runs per scenario, so it's not something you can count on.
Both models, though, hit an identical wall: config correctness sat at 0% across every SPFx scenario for both. Neither model could invent the seven specific structural changes that simply weren't documented anywhere. That's the real finding buried under all the token math: when the underlying content has gaps, no model version fixes it. And token variance made things worse for Sonnet 5, which swung from 16,000 tokens to 6.6 million on the exact same architecture scenario, while Sonnet 4.6 stayed clustered tightly between 14,000 and 45,000 tokens, making it far easier to budget against.
The takeaway the team lands on is blunt: treat a model upgrade as a hypothesis, not a fact, and test it against your own workload before switching. Sonnet 5 earned its keep on instruction-following upgrade tasks and lost ground on routine architecture work. Which one wins depends entirely on what you're actually asking it to do.
My take — AI-written commentary, not fact-checked reporting
Rate cards are marketing, not budgets, and anyone switching models purely because the sticker price dropped 33% deserves the surprise bill they get. The real story here isn't Sonnet versus Sonnet, it's that documentation gaps beat both models equally, which should worry teams throwing model upgrades at problems that are actually content problems. Measure your own workload before you trust anyone's benchmark chart, including this one.
Read more about this at: Microsoft