TLDRocket
Sign in

vLLM added bidirectional and push-based KV cache transfer for disaggregated prefill/decode inference to reduce time-to-first-token

Feature update Provisional 84% confidence first seen

Two AWS Machine Learning articles report that vLLM introduced new mechanisms for disaggregated serving, including bidirectional KV cache transfer and a push-based KV transfer connector (via NVIDIA Inference Xfer Library) between prefill and decode components. In EC2 P5en 1P1D tests with EFA, the changes reduced TTFT and improved TPOT metrics for models such as Qwen3-32B and others, with results varying by prompt length and transfer mode.

bi-directional KV cache transfer vLLM vLLM AWS EFA Amazon EC2 P5en Amazon EC2 P5en EFA NVIDIA NVIDIA Inference Xfer Library (NIXL) NixlPushConnector Qwen3-32B zai-org/GLM-5-FP8

Source coverage

Improve TTFT for multi-turn agentic workloads with bi-directional KV transfer in vLLM Amazon Web Services Push-based KV transfer to improve TTFT and TPOT for disaggregated inference in vLLM Amazon Web Services

The daily briefing

Every AI story that matters, in your inbox by 8am.

TLDRocket reads all relevant sources, removes duplicate coverage, and summarises the day in two minutes. Follow companies and topics for alerts, or get the briefing in Slack. Free, no spam, unsubscribe anytime.