Disaggregated prefill and decode for LLM inference on SageMaker HyperPod
AWS Xuan Lu
Amazon SageMaker HyperPod now supports Disaggregated Prefill and Decode (DPD), which separates LLM inference into compute-bound prefill and memory-bound decode phases running on separate GPU pools connected via Elastic Fabric Adapter. The implementation uses vLLM with LMCache to handle long-context, high-concurrency streaming workloads, with KV cache transfer taking single-digit milliseconds on ml.p5.48xlarge instances. Organizations can now independently tune time to first token and inter-token latency while preventing long prompts from blocking concurrent decode requests.
Why it matters
In this post, we show how to implement DPD with vLLM on Amazon SageMaker HyperPod using the HyperPod Inference Operator.