Deploying Kimi K3 on Amazon SageMaker HyperPod and Amazon EKS

Moonshot AI dropped Kimi K3 on July 27, and it's not a subtle release. At 2.8 trillion total parameters, it's the first open-weight model to crack the 3-trillion-parameter class, built as a Mixture of Experts system with 896 specialists, only 16 of which fire per token. That works out to roughly 104 billion active parameters per forward pass, which Moonshot claims gives 2.5x better scaling efficiency than its predecessor, Kimi K2. AWS wasted no time publishing a deployment guide, because a model this size is basically useless without serious infrastructure behind it.

And serious is the right word. AWS says you need a p6-b300 instance packing eight NVIDIA B300 Blackwell Ultra GPUs just to run tensor-parallel inference across the full expert pool. The weights ship in MXFP4 format on Hugging Face under moonshotai/Kimi-K3, and serving them requires a special day-0 vLLM container, since support for Kimi K3's architecture hasn't been merged into the mainline vLLM build yet. This is not a model you spin up on a laptop, or honestly on most companies' existing GPU budgets.

AWS lays out two paths to get there. The first runs through SageMaker HyperPod with its Inference Operator, which handles cluster creation, model download, and health checks largely on autopilot — you apply a YAML manifest specifying the Hugging Face model ID, tensor-parallel size of 8, and a handful of Kimi-specific flags, and the operator does the rest. The second path is a self-managed Amazon EKS cluster using Terraform modules from the AI on EKS project, for teams that want more control over their Kubernetes setup and are comfortable wiring up GPU drivers and Helm charts themselves.

Both routes hit the same wall: getting your hands on p6-b300 capacity in the first place. AWS points customers toward Flexible Training Plans for HyperPod or EC2 Capacity Blocks for EKS, both of which are essentially reservation systems for scarce Blackwell Ultra GPUs. That detail says as much about the current GPU market as it does about Kimi K3 itself — the model may be open weight, but the compute to actually run it is anything but abundant.

Once deployed, the payoff is a fairly ordinary OpenAI-compatible chat completions endpoint, reachable with the same Python SDK or curl commands developers already use everywhere else. Kimi K3 supports a million-token context window, native tool calling, and an always-on reasoning mode, positioning it squarely for long-horizon coding and multi-step agent work. AWS frames this as democratizing frontier-scale AI. It's more accurate to say it's translating an open-weight release into a very expensive, very specific shopping list.