DeepSeek open-sources DeepInfer, the inference framework behind its production services

DeepSeek has put the software behind its own production inference stack on GitHub. The framework, called DeepInfer, is the engine that helps serve the company’s models at scale, and that problem is exactly how DeepSeek frames it: how do you reach a huge audience without making every request ruinously expensive?

The company says one cluster already serves 155 trillion tokens a day. It also says off-peak revenue works out to $38 per graphics processing unit per hour. Those numbers give the release a sharper edge than a normal open-source drop. This is not a toy repo. It is the plumbing for a live service.

For now, DeepInfer only supports DeepSeek-V4.1-Flash, and by default it loads the updated 1017 version DeepSeek shipped over the weekend. It runs on Nvidia’s SM100 Blackwell chips and on Huawei’s Ascend 950 neural processing units, but not on Hopper or other older Nvidia architectures. Two reproducibility features are Nvidia-only, and all of the benchmark numbers DeepSeek published were measured on Nvidia hardware too.

The framework leans on a couple of tricks to save compute. One, called DSpark, uses a draft model to propose five tokens at a time, then a confidence head decides which ones the main model will keep. DeepSeek says that cuts the number of draft tokens checked per step by about 30% and lifts throughput by 4% to 7% at its production load. Another method nearly halves total prefill work by running only the last 128 tokens of each prompt through decoder layers.

DeepSeek still hasn’t released the key-value cache management and load-balancing code. The company says those parts are tightly bound to internal infrastructure, though it does explain them in a blog post on Zhihu. Splitting prefill and decode across separate machines is also left to operators, and DeepSeek notes that most of the engine work for that is already in place. The smallest setup it calls efficient uses 64 GPUs for prefill and another 64 for decoding.

This is also a partial return to a job DeepSeek said was too much for its small research team back in April 2025, when it described the software as an early fork of vLLM built around its cluster tools. The new README says vLLM was the original foundation, and DeepInfer also uses libraries DeepSeek had already released, including FlashMLA, DeepGEMM and DeepEP. The companion deepseek-recipe library translates requests into token sequences using the same logic as DeepSeek’s official API. Future model inference code will go into the same repository.