Kimi K3 vs GPT-5.6 Sol on DeepSWE: Cost, Coding, and Routing

Benchmark wars usually produce a single winner, but the Kimi K3 vs GPT-5.6 Sol matchup on DeepSWE refuses to cooperate. Sol, the closed model, takes pass@1 with 72.7% against Kimi's 68.5%, a real but not huge 4.2-point gap. Give either model a second or fourth try, though, and the story flips. Kimi K3 edges ahead at pass@2 (82.0 vs 81.0) and pulls clear at pass@4, hitting 89.4% versus Sol's 85.8% — the best pass@4 score any flagship-tier model has posted on this benchmark.

Money makes the case sharper. Kimi K3 runs $4.65 per rollout against Sol's $8.37, and when you count solved tasks per dollar, Kimi delivers roughly 2.8 times the output — 14.7 solves per $100 compared to Sol's 5.3. The catch is time: Kimi's median rollout stretches to 66 minutes versus Sol's 17, partly because Kimi takes about 40% more steps to get there. So you're trading patience for a much smaller bill.

What's genuinely interesting is how differently these two models break. Sol is the steadier operator — it nails 61 tasks on all four attempts and only fails by wrecking a repo's existing tests 20% of the time it stumbles, a pattern Together AI says shows up across other GPT models too. Kimi K3 fails more gently: it rarely breaks baseline tests (11%) but frequently gets tantalizingly close, with 65% of its misses passing more than 80% of new tests before tripping on the rest. Reliability-wise Sol wins clearly, 84.5% four-for-four versus Kimi's 76.6%, but Kimi's coverage — solving something at least once — reaches a benchmark-best 89.4%.

Because the two models only agree on which tasks are solvable about 46% of the time, they're genuinely complementary rather than redundant copies of each other. Route Kimi first and escalate to Sol whenever your test suite rejects the output, and the cascade reaches about 85.6% accuracy — beating both models solo and even an oracle-perfect single-shot router at 83.4%. Together the pair covers 108 of 113 tasks; the remaining five stump both models entirely, meaning 95.6% is the ceiling for this duo without adding a third model to the mix.

Language-by-language, it's close to a toss-up: Sol takes Python, TypeScript, and JavaScript, Kimi wins Rust, and the two tie on Go. Sort by task type instead and clearer lanes emerge — send serialization, concurrency, and program-analysis work to Sol, hand ops tooling and runtime internals to Kimi, and flip a coin on conformance tasks. None of this makes one model obsolete. It makes the argument for treating model selection as a routing problem instead of a leaderboard contest.