Disaggregated prefill and decode for LLM inference on SageMaker HyperPod

AWS Machine Learning · 6 days ago

Amazon SageMaker HyperPod now supports Disaggregated Prefill and Decode (DPD), which separates LLM inference into compute-bound prefill and memory-bound decode phases running on separate GPU pools connected via Elastic Fabric Adapter. The implementation uses vLLM with LMCache to handle long-context, high-concurrency streaming workloads, with KV cache transfer taking single-digit milliseconds on ml.p5.48xlarge instances. Organizations can now independently tune time to first token and inter-token latency while preventing long prompts from blocking concurrent decode requests.