DeepSeek AI Released DeepSeek-V4.1-Flash with 1M Context, FP4 KV Cache, and Cross-Layer Attention Reuse
MarkTechPost Asif Razzaq ● Covered by 2 sources
DeepSeek’s new V4.1-Flash model handles 1M tokens and shrinks its cache to 890 bytes per token. That’s the whole trick: less memory pain for long-running agents.
Based on reporting by MarkTechPost, Asif Razzaq — read the original for the full story.
Summary, retelling and take written by AI under human oversight; images are AI-generated illustrations. How we work · Report an error
DeepSeek AI has pushed on the part of large-model serving that usually hurts first: memory. Its new DeepSeek-V4.1-Flash is a multimodal Mixture-of-Experts model with a 1M-token context window, but the headline is not just size. It brings the global KV cache down to 890 bytes per token, which the company says is about a quarter of DeepSeek-V4-Flash and roughly 437 times smaller than DeepSeek-V1.
The architecture is built to avoid doing the same expensive work twice. The 40-layer backbone is split into a 20-layer causal encoder and a 20-layer decoder. Prompt tokens stop at the encoder, and the decoder derives its global KV from the encoder’s final hidden state instead of computing its own. That cuts prefill work almost in half. Sliding-window attention still runs in every layer, but the decoder rebuilds those states by replaying only the last 128 prompt tokens.
DeepSeek also goes all-in on compressed sparse attention. V4.1-Flash uses CSA2, where layers are assigned Full, Reindex, or Reuse roles. Some layers build fresh indices, others rescore a shared candidate set, and many simply reuse what came before. In the decoder, the Full layer can build a candidate pool of up to 16,384 positions, so later layers score a bounded set rather than the entire context. That is the sort of engineering that makes a giant context window usable instead of theatrical.
The cache tricks extend beyond attention math. The main KV cache is quantized to E2M1 with one E4M3 scale per 16 channels, following NVFP4 without the global scale. DeepSeek says that nearly halves storage versus V4’s FP8 cache. SWA KV no longer sits on SSD either; it lives in a distributed pool carved from 10% of host DRAM, while global KV is kept for 72 hours. On a miss, only 128 tokens need to be recomputed.
Training was large-scale: 45T multimodal tokens, with sparse attention trained from scratch at 64K sequence length and context extended to 1M at 34T tokens. DeepSeek says the base model matches DeepSeek-V4-Pro-Base on world knowledge and coding while using one-third of the total parameters and one-quarter of the activated ones. Open weights are available under MIT, with vLLM, SGLang, and Transformers support on Hugging Face, plus a public API with low, high, and max reasoning tiers.
My take — AI-written commentary, not fact-checked reporting
This is the right kind of model release: less fireworks, more plumbing. The AI race keeps pretending bigger context is a simple scale-up problem, when the real fight is making caches stop eating the datacenter. Open weights under MIT just make the contrast sharper: some labs sell mystery, DeepSeek is selling a memory budget with the lid off.
Read more about this at: MarkTechPost
