Tiered KV cache for large LLMs on Amazon SageMaker HyperPod with Curvine
Amazon Web Services Qingyuan Tang
AWS and Curvine split LLM cache across GPU, CPU, and shared NVMe on SageMaker HyperPod. That can cut TTFT and let cheaper G6e boxes handle workloads that once pushed P5s.
Based on reporting by Amazon Web Services, Qingyuan Tang — read the original for the full story.
Summary, retelling and take written by AI under human oversight; images are AI-generated illustrations. How we work · Report an error
AWS is pushing a very practical fix for one of LLM inference’s least glamorous problems: the cache. If a prompt gets long, or if many requests share the same system prompt, vLLM can spend a lot of time redoing work it already did. That hurts time-to-first-token and makes bigger GPU instances look like the only safe option.
The answer here is a tiered KV cache on Amazon SageMaker HyperPod. The stack adds Curvine, a distributed cache filesystem, as a shared L2 tier on top of vLLM’s own GPU cache and CPU spill area. So the hierarchy becomes GPU HBM, then host memory, then pooled NVMe shared across replicas. In AWS’s test deployment, that reached up to a 100 percent cross-Pod cache hit rate, up to a 2.7x TTFT improvement, and about 56 ms cross-node L2 read latency for a roughly 1,900-token prompt.
The design matters because isolated replicas are the enemy of reuse. In the old setup, if a request landed on a different vLLM Pod, the cache might as well not exist. Curvine changes that by pooling the local NVMe drives on G6e and P5 instances into one namespace mounted into every inference Pod. A block written by one replica becomes readable by another, which is exactly what you want when prompts overlap.
HyperPod’s Intelligent Routing is the other half of the trick. The router can use prefix-aware, kv-aware, or round-robin strategies, and the default is built for shared prompts and multi-turn dialogue. The operator then checks GPU cache first, then CPU, then the shared NVMe pool, and only re-prefills from scratch on a full miss. AWS says workloads that previously needed P5 instances can run on lower-cost G6e instances, depending on model size and traffic.
There’s some plumbing involved, because the current CRD only accepts redis or tieredstorage natively for the L2 backend. To point at Curvine’s FUSE mount, AWS patches the LMCACHE_REMOTE_URL environment variable in the vLLM container. That’s not exactly magical, but it is the sort of slightly annoying glue work that usually decides whether a neat architecture stays a blog post or becomes something teams actually ship.
My take — AI-written commentary, not fact-checked reporting
This is the right kind of AI infrastructure work: boring, expensive, and immediately useful. The industry has spent years pretending bigger models solve everything; meanwhile, a shared cache and sane routing can do more for user patience than another shiny benchmark. Open models keep winning on flexibility, but the real moat now is operational thrift, not applause lines about scale.
Read more about this at: Amazon Web Services