Serving DeepSeek-V4: why million-token context is an inference systems problem
Together AI
DeepSeek-V4's million-token context window relies on architectural changes using compressed sparse attention, heavily compressed attention, and sliding window attention that reduce key-value cache requirements. Together achieved 3.7M tokens of capacity on an NVIDIA HGX B200 node through cache management policies, compared to 1.2M without optimization. Serving V4 efficiently requires inference engines to handle multiple cache types, implement context-aware prefix caching policies, and choose endpoint configurations matched to workload characteristics—making long-context serving primarily a systems problem rather than just a model capability.
Why it matters
DeepSeek-V4 makes million-token context a serving-systems problem. Together AI explores the inference work behind V4 on NVIDIA HGX B200, including compressed KV layouts, prefix caching, kernel maturity, and endpoint profiles for long-context workloads.
Related stories
DeepSeek AI Released DeepSeek-V4.1-Flash with 1M Context, FP4 KV Cache, and Cross-Layer Attention Reuse
MarkTechPost · 1 month ago ·
19
DeepSeek-V4.1 Flash: Pushing the Limits of KV Cache Compression
GitHub Pages · 1 month ago ·
3