Mixture-of-Experts kernel engineering on Amazon SageMaker AI
Amazon Web Services Mona Mona
AWS says it tuned MoE kernels on SageMaker AI. The payoff: up to 10% more throughput and up to 9% faster time to first token.
Based on reporting by Amazon Web Services, Mona Mona — read the original for the full story.
Summary, retelling and take written by AI under human oversight; images are AI-generated illustrations. How we work · Report an error
Amazon Web Services says it has spent time on the unglamorous part of Mixture-of-Experts inference: the kernels. That’s the right place to look. MoE models keep per-token compute low by activating only part of the network, but the routing, shuffling, and expert math still eat a big share of total runtime. If those pieces are slow, users pay for it in latency and cost.
One of the biggest fixes was a simple idea with a messy implementation: swap the A and B operands so Hopper’s WGMMA instructions see the right shape. In decode, MoE batches are tiny — often a single token row — which means the first matrix in a multiply can be too thin to fill Hopper’s fixed 64-row layout. By transposing the matrices before they hit PTX, AWS says it moved the large expert dimension into the slot Hopper likes best. That let Tensor Cores run at higher density and enabled asynchronous prefetching. The result: 2% to 4% higher end-to-end throughput on H200, and the change is upstreamed in vLLM 0.24.0 and later.
AWS also changed where FP8 MoE runs. Its profiling found that Triton beat the default FlashInfer+CUTLASS path for FP8 tensor-quantized models on Hopper, especially when batches were small and irregular. The difference came from how Triton fuses scaling, matrix multiply, and expert aggregation into one pass instead of bouncing through more rigid template-driven steps. No kernel code had to change here; this was a routing policy update. That still produced a 10% throughput gain, and AWS says the improvement is available in the LMI v28 container.
The last win was about stopping Triton from recompiling the same kernel over and over. The _fused_inv_rope_fp8_quant_per_head kernel was being rebuilt every time num_tokens changed, which is exactly the sort of overhead that turns a quick inference path into a sluggish one. Adding do_not_specialize annotations removed those recompilations and cut time to first token by 9% on H200 at 8-way tensor parallelism. AWS says this is upstream in vLLM 0.22.0 and later, and the combined optimizations now ship in the LMI Deep Learning Container on Amazon SageMaker AI.
My take — AI-written commentary, not fact-checked reporting
This is the part of AI infrastructure that actually matters: not bigger slogans, just fewer wasted cycles. The industry loves talking about model scale, but users feel kernel friction first. Open-source serving stacks keep proving that boring compiler and routing work beats another round of hype almost every time.
Read more about this at: Amazon Web Services