An Asus server rack built on Nvidia’s GB200 NVL72 Blackwell system at Computex 2024 in Taipei (illustrative). Image: 极客湾 Geekerwan / Wikimedia Commons, CC BY 3.0, cropped
_19.png){kind=link}
DeepSeek has released the software that runs its own AI service, 18 months after saying it wouldn’t. The Chinese lab published DeepInfer on GitHub on Tuesday, describing it as “the high-performance inference framework that powers DeepSeek’s large-scale production services”, and it runs on Nvidia’s Blackwell chips and Huawei’s Ascend 950 alike.
Inference is the part of AI that happens after training: taking a finished model and serving answers to millions of people quickly and cheaply. DeepSeek’s low prices have always rested on how well it does this, and DeepInfer is the first time it has shown the whole engine rather than individual parts. The code is under the Apache 2.0 licence, so anyone can use, change and sell it.
That is a turnaround. In a note in April 2025, DeepSeek said it had considered releasing its full inference engine but decided against it, because the code was a heavily customised early fork of vLLM, was “tightly coupled” with its internal cluster tools, and its small team lacked the bandwidth to maintain a big open-source project. It promised to contribute pieces to existing projects instead. DeepInfer’s README still credits vLLM as the base it was “originally built primarily on”, with ideas from SGLang and Nvidia’s TensorRT-LLM.
🚀 Introducing DeepInfer, our in-house inference framework, now fully open source!
Powering DeepSeek-V4.1-Flash at scale with high throughput and low latency. We’re sharing our production stack to help the community lower compute costs, reduce energy use, and make AI more affordable.
🔗 github.com/deepseek-ai/DeepInfer 1/4
What developers get #
For now DeepInfer supports one model, DeepSeek-V4.1-Flash, the cheap model DeepSeek upgraded on Saturday, and it downloads the October 17 checkpoint from Hugging Face on first run. The features listed in the README are the tricks DeepSeek uses in production:
- DeepSeek’s own kernels: FlashMLA, DeepGEMM, DeepEP, TileKernels and DeepSelect, plus custom TileLang code.
- Splitting the work across chips: data and context parallelism for attention, and expert parallelism with load balancing for the model’s mixture-of-experts layers.
- Faster decoding: a system called DSpark drafts several tokens ahead and uses confidence estimates to decide when to check them.
- Repeatable answers: results that stay identical whatever else is in the batch, and deterministic sampling with a fixed seed, both on Nvidia only.
- Structured outputs: answers forced to match a regex, JSON schema or grammar.
DeepSeek’s benchmark charts were measured on 4 GPUs for prefill and 64 for decode. It needs Linux, Python 3.12 and either CUDA 13 or Huawei’s CANN 9.2.
What isn’t included #
This is an offline inference release, and DeepSeek is upfront that it is not a full service. The README lists three things a complete deployment needs that are missing: a KV cache system, which stores earlier parts of conversations so they don’t have to be recomputed; splitting prefill and decode onto separate machines, though it says “most required engine functionality is already implemented”; and load balancing across replicas. Those are the pieces that turn a fast engine into a cheap public API, so companies hoping to copy DeepSeek’s pricing will still have to build them or borrow them from vLLM or SGLang.
Nvidia and Huawei, side by side #
The hardware list has two entries: Nvidia’s SM100 GPUs, the Blackwell data-centre generation, and Huawei’s Ascend 950 NPUs. It follows DeepSeek’s release last month of Ascend versions of its core libraries, and its founder’s push to train its biggest models on Chinese chips. A production-grade engine that treats the Ascend as a first-class target is the kind of software Huawei has lacked against Nvidia’s CUDA.
It also lands the day after the US moved to ban DeepSeek from every federal civilian agency, an order that covers “third-party products and services developed using DeepSeek implementations”. The order does not name products or say whether open code counts, so any US vendor building on DeepInfer and selling to government will want an answer first. Our guide to where DeepSeek is banned covers the rest of the world.
Why it matters #
DeepSeek has handed out the recipe behind its rock-bottom prices, minus the parts that make it a service. That lowers the bar for anyone serving its models cheaply on Nvidia or Huawei hardware, even as Washington tries to keep DeepSeek out of government.
Sources: DeepSeek, DeepInfer on GitHub; DeepSeek on X; DeepSeek, “The Path to Open-Sourcing the DeepSeek Inference Engine”.