Tiered KV cache for large LLMs on Amazon SageMaker HyperPod with Curvine

Amazon Web Services · 2 months ago · 24

Amazon SageMaker HyperPod with vLLM’s LMCache was extended using Managed Tiered KV Cache plus Intelligent Routing, with Curvine added as a shared NVMe (L2) layer to reuse KV cache across Pods and nodes. In a test deployment, it reported up to a 100% cross-Pod cache hit rate, up to a 2.7x improvement in TTFT, and about 56 ms cross-node L2 read latency for a ~1,900-token prompt. Workloads can shift from requiring P5 instances to running on lower-cost G6e instances by reducing cold-start re-prefill penalties when prompts share leading tokens.