Mixture-of-Experts kernel engineering on Amazon SageMaker AI

Amazon Web Services · 1 day ago · 23

Amazon SageMaker AI described kernel-level optimizations for fused mixture-of-experts (MoE) inference on NVIDIA H200 via vLLM, including a Hopper-specific fused-MoE change, backend routing tweaks, and Triton JIT recompilation fixes. The launch reports a 2–4% end-to-end throughput improvement on H200 from the “swap-AB for WGMMA utilization” optimization. These changes are integrated into AWS’s Large Model Inference container on SageMaker AI and into vLLM releases (for example, 0.24.0+ and 0.22.0+), improving throughput and time-to-first-token.