Cache-aware prefill–decode disaggregation (CPD) for up to 40% faster long-context LLM serving
Together AI
Together AI developed cache-aware prefill-decode disaggregation (CPD), a serving architecture that separates requests with reusable cached context from those requiring full computation, improving long-context LLM inference efficiency. The system achieves up to 40% higher sustainable throughput and lower time-to-first-token compared to standard disaggregated designs by routing warm requests (those with cache hits) separately from cold requests (those with new context). The architecture uses a three-tier KV-cache hierarchy and intelligent routing to prevent expensive cold prefills from blocking fast paths for requests that can reuse previously computed context.
Why it matters
Serving long prompts doesn't have to mean slow responses. Learn how Together AI's CPD architecture separates warm and cold inference workloads to deliver 40% higher throughput and dramatically lower time-to-first-token for long-context LLM serving.
