Push-based KV transfer to improve TTFT and TPOT for disaggregated inference in vLLM
Amazon Web Services Sunita Nadampalli ● Covered by 2 sources
AWS added push-based KV transfer to vLLM. It cuts first-token and per-token latency by moving setup off the decode path.
Based on reporting by Amazon Web Services, Sunita Nadampalli — read the original for the full story.
Summary, retelling and take written by AI under human oversight; images are AI-generated illustrations. How we work · Report an error
AWS says it has pushed a new KV-transfer mode into vLLM that trims the slowest part of disaggregated inference: the handoff between prefill and decode. In the setup vLLM already supports, one engine builds the key-value cache from the prompt, and another uses it to generate tokens. The catch is that the cache has to move between them fast enough that the transfer itself doesn’t become the bottleneck.
The old pull flow made the decode side wait for several serial steps after prefill finished: the router relayed metadata, the decode instance allocated blocks, and then it kicked off an NIXL READ to pull the cache. Push mode changes that sequence. The decode side registers its target blocks earlier, while prefill is still working, and a background thread on the prefill side handles the actual NIXL WRITE once both sides are ready.
That matters because it removes work from the critical path to the first token. AWS says the push connector overlaps decode-side setup with prefill, rather than making the decode worker do that setup after prefill completes. The design also lets the prefill side release KV blocks sooner, although it does force the decode instance to be chosen earlier, which is less flexible than the pull model for load-aware routing.
In AWS’s benchmark, the company used Qwen3-32B on a 1P1D deployment with Amazon EC2 P5en instances connected with EFA, and it disabled prefix caching on the decode node so every request had to fetch the full KV cache. Under those conditions, push mode reached roughly 540 ms TTFT at 8 QPS and 16k input length, while pull mode climbed to about 1,744 ms. TPOT followed the same pattern: push stayed near 7 ms, while pull rose to about 10.4 ms at heavier load.
AWS says the feature was built and validated on Trainium clusters and then contributed back to vLLM as accelerator-agnostic code. There’s also a follow-on idea on the table: bring the same background-thread trick to pull mode too, which is the sort of obvious cleanup that usually shows up after someone has already made the fast path work the hard way.
My take — AI-written commentary, not fact-checked reporting
This is the kind of engineering that actually deserves the word optimization. Not bigger models, not louder demos, just moving the boring plumbing off the hot path so the machine can stop tripping over its own feet. The broader pattern is simple: inference wins now come from system design, not marketing swagger.
Read more about this at: Amazon Web Services