Google Cloud Ships llm-d 1.0 With Cache-Aware Routing for TPU and GPU Fleets
MarkTechPost Asif Razzaq ● Covered by 3 sources
Google released llm-d 1.0, a stable inference stack for Kubernetes. It routes around cache misses and queue jams, which can cut TTFT and waste less GPU/TPU time.
Based on reporting by MarkTechPost, Asif Razzaq — read the original for the full story.
Summary, retelling and take written by AI under human oversight; images are AI-generated illustrations. How we work · Report an error
Google has shipped llm-d 1.0, the first stable release of its distributed inference stack, and it is aiming squarely at the messy part of model serving: keeping self-hosted agents and RL jobs moving when round-robin load balancing starts to fall apart. The release landed at PyTorch Conference North America.
llm-d sits on Kubernetes and turns vLLM or SGLang servers into one coordinated pool. It runs across NVIDIA, AMD and Intel GPUs, plus Google TPUs. Under the hood, the big ideas are cache-aware routing, tiered KV cache offload, and splitting prefill from decode so each part of the workload can be handled where it makes the most sense.
The headline benchmark is a 17,000-node, multi-region GKE deployment that Google says hit 2.9x throughput scaling with 99.9% success and under 1% overhead. On an Ironwood TPU coding workload, median time to first token dropped by more than 85%, while throughput rose 2.4x versus a plain Kubernetes Service. Google also says its Gemini platform doubled prefix cache hit rates from 35% to 70%.
Version 1.0 is also about making the stack usable in production, not just clever in demos. The new bits include priority and fairness-aware flow control, router high availability with hot standby, DisaggregatedSet for rolling prefill and decode pools together, session-aware routing, turn-priority scheduling and native Responses API state.
There is also a more practical angle here: llm-d-async-processor can pull jobs from Google Cloud Pub/Sub or Redis, and TorchTPU lets PyTorch models run natively inside upstream vLLM and SGLang. Google says the first stable release builds on a project with more than 500 contributors from over 25 organizations, and that its published runs can be compared on Prism.
My take — AI-written commentary, not fact-checked reporting
The real story is not that Google built another serving stack. It is that the industry is finally admitting inference is a scheduling problem with a memory problem attached. Fancy model weights are useless if your cache is in the wrong place and your accelerators are sitting around twiddling their thumbs. llm-d looks like the boring infrastructure work that actually matters, which is usually where the money goes after the demo fog clears.
Read more about this at: MarkTechPost