Deploying Kimi K3 on Amazon SageMaker HyperPod and Amazon EKS

Open-weight models keep getting bigger, and Kimi K3 is the biggest yet. Moonshot AI dropped it on July 27, 2026, and at 2.8 trillion total parameters it's the first open-weight system to crack the 3 trillion parameter tier. That's a staggering number to put on Hugging Face for anyone to download, and it immediately raises the obvious question: who actually has the infrastructure to run something this large?

AWS answered that question with a walkthrough, not a press release. Kimi K3 uses a Mixture of Experts setup with 896 specialist experts, but only 16 fire on any given token, which keeps active parameters down to around 104 billion per forward pass. Moonshot says that architecture, combined with things like Kimi Delta Attention and a Stable LatentMoE framework, gets a 2.5x efficiency gain over the previous Kimi K2 model. The weights ship in MXFP4 format, a 4-bit scheme meant to balance quality against memory footprint, and serving requires a day-0 vLLM container built specifically for this model since the changes haven't landed in the main vLLM release yet.

None of this runs on modest hardware. AWS specifies the ml.p6-b300.48xlarge instance, packing 8 NVIDIA B300 Blackwell Ultra GPUs, as the minimum viable unit for tensor-parallel inference across the full expert pool. Getting that capacity isn't trivial either, so AWS points customers toward two reservation mechanisms: Flexible Training Plans for SageMaker HyperPod clusters, and EC2 Capacity Blocks for teams running their own EKS setup. Both exist because B300 capacity is the kind of thing you reserve ahead of time, not something you spin up on demand.

The deployment paths themselves diverge based on how much control a team wants. HyperPod with the Inference Operator is the managed route — create a cluster through the SageMaker console, attach a training plan for capacity, then apply a single InferenceEndpointConfig manifest that handles model download from Hugging Face, container scheduling, and health checks automatically. The EKS path is more hands-on: provision the cluster via Terraform, reserve GPUs through a Capacity Block, install NVIDIA drivers and the device plugin, then deploy vLLM yourself with a tensor-parallel size of 8 and the same MXFP4 arguments. Either way, the endpoint that comes out the other end is OpenAI-API compatible, so switching an existing application over is mostly a matter of pointing it at a new base URL.

What's notable here isn't the tooling itself — HyperPod and EKS have hosted large models before — it's that Moonshot chose to open-weight something at this scale at all. A 2.8 trillion parameter model with a million-token context window and native multimodal support is the kind of thing that used to stay locked behind an API. Now it's a Hugging Face model ID and a Kubernetes manifest away from running on your own account, provided you can get your hands on eight Blackwell Ultra GPUs.