Cache-aware prefill–decode disaggregation (CPD) for up to 40% faster long-context LLM serving

Together AI · 4 months ago

Together AI developed cache-aware prefill-decode disaggregation (CPD), a serving architecture that separates requests with reusable cached context from those requiring full computation, improving long-context LLM inference efficiency. The system achieves up to 40% higher sustainable throughput and lower time-to-first-token compared to standard disaggregated designs by routing warm requests (those with cache hits) separately from cold requests (those with new context). The architecture uses a three-tier KV-cache hierarchy and intelligent routing to prevent expensive cold prefills from blocking fast paths for requests that can reuse previously computed context.