Kimi K3 vs GPT-5.6 Sol on DeepSWE: Cost, Coding, and Routing

Two coding models, two different personalities. That's the takeaway from Together AI's head-to-head between Kimi K3 and GPT-5.6 Sol on DeepSWE, a benchmark that grades software engineering across many task types and languages. Sol, tested two weeks ago, holds the pass@1 crown with 72.7% versus Kimi K3's 68.5%, a 4.2 point edge on a single attempt. It's also the more dependable model, nailing 61 tasks on all four tries compared to Kimi's 45.

But give either model more than one shot and the story flips. Kimi K3 catches up at pass@2 (82.0 vs 81.0) and pulls ahead at pass@4, hitting 89.4% against Sol's 85.8%. And it does that while costing $4.65 per rollout versus Sol's $8.37 — Together AI counts that as roughly 2.8 times more solved tasks per dollar. The tradeoff shows up in time: a median Sol rollout finishes in 17 minutes, Kimi's takes 66, using notably more steps along the way.

Breaking pass@1 and pass@4 into coverage and reliability shows just how differently these two models operate. Sol is the steady hand — 84.5% reliability, but a somewhat narrower 85.8% coverage. Kimi K3 covers more ground, solving 89.4% of tasks at least once, but backs off on consistency at 76.6% reliability. The two models only agree on which tasks they solve about 0.46 of the time, a low correlation that Together AI contrasts with an earlier Kimi K3 comparison against Fable, where the models were near-identical at 0.72.

That divergence is exactly what makes routing between them useful. Combined, Kimi K3 and Sol solve 108 of the 113 tasks, a 95.6% ceiling that Together AI says would require a third model to push past. The practical setup: run Kimi K3 first, and only escalate to Sol when a test suite rejects the output. That cascade lands around 85.6% — beating both models solo and even a theoretical perfect one-shot router at 83.4% — while staying cheaper than running Sol alone, since Kimi clears an estimated 70% of the workload before Sol ever gets called in.

The failure patterns diverge too. Sol tends to break the repository's existing tests in about 20% of its misses, which Together AI notes lines up with other GPT models. Kimi K3 rarely breaks baseline tests, at 11%, but often lands close without fully passing — 65% of its failures were near misses where over 80% of new tests succeeded. Per programming language, the split is granular: Sol leads Python, TypeScript, and JavaScript, Kimi takes Rust, and the two tie on Go.