Cache-aware prefill–decode disaggregation (CPD) for up to 40% faster long-context LLM serving

Long prompts are quietly becoming the default in AI apps. Coding assistants that read your whole repo, chat agents that remember five turns back, retrieval systems stuffing in whole documents — none of that works well if the server treats every request as brand new. Together AI's answer is something they call cache-aware prefill-decode disaggregation, or CPD, and the core insight is almost embarrassingly simple: stop letting a 100,000-token cold prompt sit in the same queue as a request that's 90% identical to something the system just processed.

Most serving stacks already split prefill (the expensive part where the model reads your prompt) from decode (the part where it generates tokens). That helps, but it doesn't solve the problem of prefill capacity itself getting clogged. If ten users send huge, unseen prompts at once, they can occupy the prefill GPUs for seconds, and anyone else waiting behind them — even someone asking a trivial follow-up in an existing conversation — gets stuck too. Together's fix adds a third tier: pre-prefill nodes that absorb the genuinely new, expensive work, while separate prefill nodes handle requests where most of the context can be pulled from a shared KV cache instead of recomputed from scratch.

That cache lives across three layers — GPU memory, host RAM, and a cluster-wide store reachable over RDMA — so a context that took seconds to compute the first time can be fetched in a few hundred milliseconds the second or third time it shows up. In their coding-agent benchmark, run on Nvidia B200 GPUs, the baseline setup topped out around 0.75-0.8 queries per second per GPU before latency exploded into multi-second territory. CPD kept scaling to roughly 1.1-1.15 QPS per GPU under the same load, about a 40% bump in sustainable throughput, while keeping median time-to-first-token in the sub-second to low-second range even as the old system was already drowning.

What's notable is that tail latency didn't get worse to buy that median improvement — p90 TTFT stayed comparable or better than the baseline across the load range they tested. And when they doubled decode capacity, the gains held up rather than disappearing, meaning this isn't some narrow trick that only works in one configuration. Generation throughput barely changed between the two setups, which tells you the entire win comes from smarter prefill scheduling, not some hidden decode optimization.

The bigger point Together is making, deliberately or not, is that as context windows keep growing, the bottleneck stops being raw FLOPs and starts being how well a system recognizes and reuses what it's already computed. That's a scheduling and systems problem as much as a modeling one, and it's going to matter more as agentic workflows push average prompt sizes further up.