DeepSeek pushed DeepInfer to GitHub this morning under Apache-2.0 — not a trimmed-down research release, but the actual inference engine running their V4.1-Flash production API. It is the most transparent look yet at how a frontier AI lab serves a large MoE model at scale. There is a catch: it targets Nvidia SM100 GPUs (the B200/GB200 generation) and Huawei Ascend 950 NPUs, requires a minimum of 128 chips in a prefill-plus-decode split, and ships without the KV cache system, the load balancers, or the online serving layer. You get the engine. Building the car is still your problem.
What DeepInfer Actually Does #
DeepInfer started life as a fork of vLLM, then diverged heavily with ideas pulled from SGLang and Nvidia’s TensorRT-LLM. The result is a single-model inference stack purpose-built for V4.1-Flash, and it ships with capabilities that most open-source frameworks are still catching up to:
- DSpark speculative decoding: Confidence-scheduled drafting — the engine generates several candidate tokens ahead and verifies them in a single forward pass, cutting latency without sacrificing accuracy.
- MoE expert parallelism: Attention data-parallel, balanced context-parallel, and expert-parallel execution with load balancing baked in — necessary at this scale for DeepSeek’s mixture-of-experts architecture.
- Zero-overhead async scheduling: The GPU runs step N while the CPU schedules step N+1, eliminating the wait that degrades throughput on naive implementations. DeepSeek credits this approach to SGLang and NanoFlow.
- Structured outputs: Constrained generation via regex, JSON schema, or EBNF — useful for production API workloads where you need predictable output shape.
The code is the real thing. That matters for reproducibility: DeepSeek’s throughput numbers can now be verified independently, and the architectural decisions are all visible. The New Stack’s analysis notes that DeepInfer gives the industry a reference implementation for MoE inference at a scale few labs have published before.
What Is Not Included #
DeepSeek is explicit about the gaps in the deployment Q&A doc, which is worth reading before you start planning. Three things that make a production inference service are absent:
- KV cache system: No distributed cache, no prefix reuse across requests over time. DeepSeek’s internal stack uses a 3FS-backed store; none of that ships. You will need to build or bolt on your own solution.
- Prefill/decode disaggregation orchestration: The engine supports the two-phase split architecturally, but the orchestration layer that moves requests and KV tensors between prefill and decode pools is tied to internal infrastructure and was not released.
- Load balancers: DeepSeek runs separate load balancers for prefill, decode, vision encoding, and expert parallelism. Only the expert-parallel load balancer ships. The rest are marked as internally coupled.
This is not a complaint — these components are genuinely hard to decouple from custom infrastructure. But any developer planning to build a production service on top of DeepInfer should price in significant engineering work before the first API call goes live.
The Hardware Requirement Is Not Negotiable #
Nvidia SM100 means B200 and GB200 silicon — the current generation that most organizations do not have access to yet. Huawei Ascend 950 is the alternative. Either way, the minimum viable deployment unit is 64 prefill GPUs plus 64 decode GPUs. If your infrastructure tops out at H100s or A100s, DeepInfer does not run on it.
The software requirements follow: Linux, GCC 13 or later, Python 3.12, and either CUDA 13 for Nvidia or CANN 9.2 for Ascend. None of these are exotic, but they all demand recent toolchain versions that some production environments have not yet standardized on.
The Huawei Ascend Angle #
Supporting Huawei Ascend 950 on equal footing with Nvidia in a major open-source ML release is worth noting on its own. DeepSeek stated last week that it plans to run pre-training on Ascend hardware by the end of 2026. DeepInfer with first-class Ascend support is confirmation that this plan is operational, not aspirational. It is also the first production-grade open-source LLM inference stack to treat Chinese silicon as a primary target — a signal that carries weight beyond the technical details, particularly given the current state of US-China chip export controls.
What Developers Without 128 SM100s Can Do #
The hardware wall is real, but the codebase still has value for developers who will never run it. DSpark’s speculative decoding implementation, the async scheduling design, and the MoE parallelism strategy are all readable and documented well enough to learn from. If you work on inference infrastructure — even for smaller models on more accessible hardware — DeepInfer is a useful reference for how a team running this at scale has approached these problems.
For context on the model itself, see the earlier coverage of DeepSeek V4.1-Flash-1017. DeepInfer is available now at github.com/deepseek-ai/DeepInfer under Apache-2.0.