Moonshot AI Releases Kimi K3: A 2.8 Trillion Parameter Open MoE Model With Kimi Delta Attention and 1M Context
MarkTechPost Asif Razzaq ● Covered by 7 sources
Moonshot AI dropped Kimi K3, a 2.8-trillion-parameter open model with a 1-million-token context window. It's the biggest open model ever released, though it still trails Claude and GPT flagships.
Based on reporting by MarkTechPost, Asif Razzaq — read the original for the full story.
Summary, retelling and take written by AI under human oversight; images are AI-generated illustrations. How we work · Report an error
Moonshot AI has a habit of quietly redefining what 'open model' means, and Kimi K3 keeps that streak going. At 2.8 trillion parameters, it's the largest openly released model anyone has shipped, and Moonshot is calling it the world's first open 3T-class system. For nine of the last twelve months, a Kimi model has held the record for biggest open weights out there. This one adds native vision and a context window stretching to a million tokens, aimed squarely at long-form coding and multi-step reasoning tasks that choke smaller models.
What's actually interesting isn't the parameter count, it's how Moonshot got there without the training and serving costs spiraling out of control. Two new mechanisms do the heavy lifting. Kimi Delta Attention, a hybrid linear attention setup, claims up to 6.3x faster decoding at million-token context lengths. Attention Residuals works on the depth axis instead, selectively pulling representations from earlier layers rather than piling them up uniformly, which Moonshot says buys about 25% more training efficiency for under 2% extra compute cost.
Sparsity is where things get aggressive. K3 runs on a Stable LatentMoE setup that activates just 16 of 896 experts at any given time. That's an extreme ratio, and it forces some genuinely new engineering to keep training stable. Moonshot's answer is Quantile Balancing, which derives expert routing straight from router-score quantiles instead of relying on brittle heuristic tuning. They also extended the Muon optimizer to tune attention heads independently, and added a couple of smaller tweaks — Sigmoid Tanh Unit and Gated MLA — to sharpen activation control and attention selectivity. Combined with updated training data recipes, Moonshot claims roughly 2.5x better scaling efficiency compared to the previous Kimi K2.
None of this matters if the model is impossible to run, so Moonshot built serving considerations in from the start. K3 uses quantization-aware training beginning at the supervised fine-tuning stage, landing on MXFP4 weights paired with MXFP8 activations so it plays nice with a wider range of hardware. Moonshot recommends supernode setups with 64 or more accelerators for anyone actually deploying this thing. And because the new attention mechanism breaks assumptions baked into existing prefix-caching code, Moonshot went ahead and contributed a fix directly to vLLM rather than leaving it to the community to figure out.
Moonshot isn't pretending K3 beats the top proprietary systems. By their own admission, it still trails Claude's and OpenAI's current flagship models on overall capability, even while topping their internal eval suite against other open and closed competitors they tested. That's a refreshingly honest framing for a company that just shipped the largest open model in existence — bragging about scale while being upfront about where the ceiling still sits.
My take — AI-written commentary, not fact-checked reporting
I like that Moonshot didn't dress this up as a GPT-5 killer, because it isn't, and pretending otherwise is how you lose credibility fast in this space. What actually matters here is the engineering: 16-of-896 expert sparsity plus quantization-aware training from day one is the kind of unglamorous plumbing work that makes trillion-parameter open models something you can actually run, not just cite in a paper. That's the real gift to the open-source ecosystem, not the parameter count.
Read more about this at: MarkTechPost