Serving DeepSeek-V4: why million-token context is an inference systems problem

Together AI · 2 months ago

DeepSeek-V4's million-token context window relies on architectural changes using compressed sparse attention, heavily compressed attention, and sliding window attention that reduce key-value cache requirements. Together achieved 3.7M tokens of capacity on an NVIDIA HGX B200 node through cache management policies, compared to 1.2M without optimization. Serving V4 efficiently requires inference engines to handle multiple cache types, implement context-aware prefix caching policies, and choose endpoint configurations matched to workload characteristics—making long-context serving primarily a systems problem rather than just a model capability.