Deploying Kimi K3 on AWS

Open-weight models keep getting smarter, but that intelligence comes with a size problem. Moonshot AI's new Kimi K3, released July 27, 2026, is a case study in that tradeoff: it's a 2.8 trillion parameter Mixture of Experts model, the first open-weight system to break into the 3 trillion parameter class, and AWS just published a detailed walkthrough for actually hosting the thing.

The architecture is what makes K3 tick. It spreads its parameters across 896 specialist experts but only activates 16 per token, meaning roughly 104 billion parameters actually fire during any given forward pass. Moonshot says that setup delivers a 2.5x improvement in scaling efficiency compared to its predecessor, Kimi K2. The model also carries a 1 million token context window, native multimodal support for text and vision, and features like always-on thinking mode and structured tool calling that target long-horizon coding and agentic workflows.

Getting weights is the easy part — they're on Hugging Face under moonshotai/Kimi-K3, distributed in MXFP4 format to balance quality against memory footprint. Serving them is where things get demanding. AWS specifies you need a p6-b300 instance, the ml.p6-b300.48xlarge, which packs 8 NVIDIA B300 Blackwell Ultra GPUs, and a day-0 vLLM inference container built specifically for K3's MoE architecture and quantization format. This is not something you spin up on a spare GPU.

AWS lays out two deployment paths. One runs through SageMaker HyperPod with its Inference Operator, which handles container orchestration and model loading automatically once you've provisioned a cluster and secured GPU capacity via a Flexible Training Plan. The other is a self-managed Amazon EKS cluster, using EC2 Capacity Blocks to reserve p6-b300 instances for a fixed window rather than committing long-term. Both routes converge on the same vLLM configuration underneath, exposing an OpenAI-compatible chat completions endpoint once everything's running.

What's notable here isn't just that K3 exists, but that deploying a model this size now has a documented, semi-standardized path on major cloud infrastructure. Whether you want AWS to manage the orchestration or you'd rather run your own Kubernetes setup, the underlying requirement is the same: serious Blackwell-class GPU capacity, reserved in advance, because this is not hardware you find lying around on demand.