Improve TTFT for multi-turn agentic workloads with bi-directional KV transfer in vLLM
Amazon Web Services Sunita Nadampalli ● Covered by 2 sources
AWS says vLLM can now send KV cache back to the prefill node, not just forward. That cuts repeat work on long agent chats and slashes TTFT on some tests.
Based on reporting by Amazon Web Services, Sunita Nadampalli — read the original for the full story.
Summary, retelling and take written by AI under human oversight; images are AI-generated illustrations. How we work · Report an error
AWS is trying to fix a very specific annoyance in multi-turn agent workflows: every new reply can make the system reprocess a growing pile of old context. In a prefill-decode setup, that means the same conversation history gets worked over again and again. The result is predictable. TTFT creeps up as the chat gets longer.
The company’s answer is bi-directional KV transfer in vLLM. Instead of only moving key-value cache from prefill to decode, the decode node can hand previously computed KV blocks back to prefill on later turns. That lets the next request reuse work it already paid for, rather than rebuilding the whole prefix from scratch. AWS says the feature was first built and validated on Trainium clusters connected with EFA, then contributed to vLLM as an accelerator-agnostic capability.
This is not just a connector tweak. AWS extended the NVIDIA Inference Xfer Library KV connector so prefill nodes can consume remote KV blocks, and it added a router that knows which conversation maps to which decode-side metadata. The router keeps only block-location metadata in memory, not the tensors themselves. Decode also has to keep blocks around for a while after generation, using a configurable TTL. The default for that is 480 seconds. There is also a recompute threshold, set to 64 tokens by default, because sometimes pulling KV over the network is slower than just recomputing a short prefix locally.
The payoff shows up in AWS’s 1P1D benchmark on Amazon EC2 P5en instances with EFA. With bi-directional transfer on, TTFT stayed much flatter as prompts grew. For Qwen3-32B, TTFT went from 122 ms to 460 ms across the tested range, compared with 158 ms to 778 ms when the feature was off. For zai-org/GLM-5-FP8, it went from 358 ms to 702 ms versus 500 ms to 2,334 ms without KV reuse. At the longest prompt lengths, AWS says that works out to as much as a 1.6x TTFT reduction for Qwen3-32B and 3.3x for GLM-5-FP8.
The interesting part is not the fancy acronym salad. It is that long-lived agent chats are finally being treated like stateful systems instead of expensive amnesiacs. That is the right direction, and it is also a reminder that a lot of ‘latency problems’ are really just self-inflicted recomputation.
My take — AI-written commentary, not fact-checked reporting
This is the sort of plumbing work that matters far more than another glossy model demo. AI vendors love talking about bigger models; the boring truth is that keeping context alive without recomputing it is where real systems get faster. The industry keeps reinventing cache management like it’s a philosophical breakthrough.
Read more about this at: Amazon Web Services