[AINews] DeepSeek DeepInfer: Open Source SOTA Inference running 155T toks/day/cluster making $22m/day at ~80% margin for a 81 day payback
Latent Space ● Covered by 10 sources
DeepSeek open-sourced DeepInfer, its production inference stack. It claims huge cached-token throughput and an 81-day payback, but not the whole serving system is out yet.
Based on reporting by Latent Space — read the original for the full story.
Summary, retelling and take written by AI under human oversight; images are AI-generated illustrations. How we work · Report an error
DeepSeek has opened up DeepInfer, the production inference stack behind V4.1 Flash, on the first day of PyTorch Conf. That’s the part of the release that matters most. The code is out, but so is the fine print: DeepSeek says some of the highest-value pieces, including KV-cache management and load balancing, are still tied to internal infrastructure.
The stack is built around distributed inference, speculative decoding, SWA Bounded Replay, and structured output on both NVIDIA and Huawei Ascend. It also includes deepseek-recipe, which handles production token encoding and decoding and aims to match the official API. DeepSeek says future model inference work and optimizations will land in the same repository too.
The numbers are the part everyone is arguing over. DeepSeek’s own report says one cluster is serving 155 trillion tokens a day, with 96% input-cache hits, 400 ms average time to first token, and 230 tokens per second for generation, rising to 300 tokens per second for coding. The same material says that works out to about $38 of revenue per accelerator-hour at off-peak prices. The catch is obvious: this is a heavily cached workload, not a clean measure of fresh token generation.
The architecture is built to make cache reuse the star of the show. Persistent block KV lives in a separate 3FS storage cluster with roughly 100 PiB of SSD capacity, and retention runs three days for agent traffic and seven for chatbot traffic. Any prefill worker can fetch a cached prefix, so compute placement and cache locality do not have to line up neatly. DeepSeek also says SWA Bounded Replay can rebuild short-lived state from only the latest 128 tokens, cutting prefill compute nearly in half.
There’s a broader industry fight hiding inside the release. DeepSeek is pushing the idea that open source can run production inference at this scale, while vLLM has already pushed back with its own reproduction results, including decode throughput within 6% of DeepInfer under a matched setup on GB200. So this is not just a code drop. It’s a shot across the bow of the closed serving shops, and a reminder that the economics of cached tokens are doing a lot of the heavy lifting here.
My take — AI-written commentary, not fact-checked reporting
This is exactly the kind of release that separates real infra from demo theatre. Open source gets the engine, while the secret sauce stays in the bits that make the bill look magical; very efficient, very on-brand. The industry keeps pretending serving is a model problem when it’s really a caching, routing, and billing problem with better branding.
Read more about this at: Latent Space
