DeepSeek Open-Sourced DeepInfer. It's a vLLM Fork, and That's the Interesting Part

DeepSeek open-sourced DeepInfer, an Apache 2.0 inference engine for serving its models, which quickly passed 16k GitHub stars and topped Hacker News. The engine is a fork of vLLM with DeepSeek's own kernel stack and ideas borrowed from SGLang and TensorRT-LLM, initially supporting only DeepSeek-V4.1-Flash across NVIDIA and Ascend accelerators. It adds batch-invariant inference and deterministic sampling modes plus built-in structured output, but ships as offline inference without the disaggregation pieces needed for production serving, and no independent benchmarks against vLLM or SGLang exist yet.

DeepSeek open-sourced DeepInfer , the inference engine that serves its production models. It hit the top of Hacker News with 2,000+ points and the repo passed 16k stars almost immediately. Hype aside, what did they actually ship? I read the repo so you don't have to. An Apache 2.0 inference framework for running DeepSeek models across lots of accelerators. Initial support is exactly one model: DeepSeek-V4.1-Flash . Hardware targets: The README is refreshingly honest about lineage: it was "originally built primarily on vLLM," with ideas borrowed from SGLang and TensorRT-LLM. So this is not a clean-sheet engine. It's a vLLM descendant with DeepSeek's kernel stack bolted on. The kernel list reads like a greatest-hits of DeepSeek's open-source drops: On top of that: Two flags matter more than any throughput chart: Same output regardless of what else is in the batch DEEPINFER BATCH INVARIANT=1 Deterministic sampling with a fixed seed DEEPINFER BATCH INVARIANT=1 DEEPINFER ENABLE MTP VARLEN=0 Batch-invariant inference means your output doesn't change because some other user's request landed in the same batch. If you've ever tried to debug an eval that flakes at temperature 0, you know why this is gold. Most engines give you "mostly deterministic." This one says it has a mode that guarantees it. Verify that claim on your workload before you bet on it. Structured output regex, JSON schema, EBNF is also built in. Requirements are not casual: Linux, GCC 13+, Python 3.12, CUDA 13 or CANN 9.2 for Ascend . export DEEPINFER ACCELERATOR=nvidia ./build.sh python examples/offline inference.py Point it at local weights with DEEPINFER HF MODEL DIR=/path/to/DeepSeek-V4.1-Flash-1017 . This is not a drop-in serving stack. The README says the release is offline inference with the core features for serving at scale, and explicitly leaves out: Those are exactly the pieces that make or break production serving. DeepSeek ships a companion "deepseek-recipe" to reproduce their API's token processing, but you're still assembling the rest yourself. Benchmarks are Pareto charts prefill on 4 SM100s, decode on 64 in the repo's images folder. I found no independent head-to-head against vLLM or SGLang yet. Treat any "X times faster" tweet as unverified until someone reproduces it. Those two engines have traded the lead for two years, so a single chart proves little. The value isn't a vLLM killer. It's that a frontier lab published the serving layer, not just weights. Weights without the engine are half a release, because the engine is where the cost per token lives. The stated goal is to help the community cut compute costs and energy use, and the API-pricing pressure that follows is the real story for anyone paying for tokens. Watch for two things: someone running independent benchmarks, and the missing disaggregation pieces landing. Until then, it's a seriously impressive reference implementation, not a turnkey replacement. Sources: the DeepInfer GitHub repo, The Decoder's coverage, and the Hacker News thread.