{"slug": "deepseek-open-sourced-deepinfer-it-s-a-vllm-fork-and-that-s-the-interesting-part", "title": "DeepSeek Open-Sourced DeepInfer. It's a vLLM Fork, and That's the Interesting Part", "summary": "DeepSeek open-sourced DeepInfer, an Apache 2.0 inference engine for serving its models, which quickly passed 16k GitHub stars and topped Hacker News. The engine is a fork of vLLM with DeepSeek's own kernel stack and ideas borrowed from SGLang and TensorRT-LLM, initially supporting only DeepSeek-V4.1-Flash across NVIDIA and Ascend accelerators. It adds batch-invariant inference and deterministic sampling modes plus built-in structured output, but ships as offline inference without the disaggregation pieces needed for production serving, and no independent benchmarks against vLLM or SGLang exist yet.", "body_md": "DeepSeek open-sourced **DeepInfer**, the inference engine that serves its production models. It hit the top of Hacker News with 2,000+ points and the repo passed 16k stars almost immediately.\n\nHype aside, what did they actually ship? I read the repo so you don't have to.\n\nAn Apache 2.0 inference framework for running DeepSeek models across lots of accelerators. Initial support is exactly one model: **DeepSeek-V4.1-Flash**. Hardware targets:\n\nThe README is refreshingly honest about lineage: it was \"originally built primarily on vLLM,\" with ideas borrowed from SGLang and TensorRT-LLM. So this is not a clean-sheet engine. It's a vLLM descendant with DeepSeek's kernel stack bolted on.\n\nThe kernel list reads like a greatest-hits of DeepSeek's open-source drops:\n\nOn top of that:\n\nTwo flags matter more than any throughput chart:\n\n```\n# Same output regardless of what else is in the batch\nDEEPINFER_BATCH_INVARIANT=1\n\n# Deterministic sampling with a fixed seed\nDEEPINFER_BATCH_INVARIANT=1 DEEPINFER_ENABLE_MTP_VARLEN=0\n```\n\nBatch-invariant inference means your output doesn't change because some other user's request landed in the same batch. If you've ever tried to debug an eval that flakes at temperature 0, you know why this is gold. Most engines give you \"mostly deterministic.\" This one says it has a mode that guarantees it. Verify that claim on your workload before you bet on it.\n\nStructured output (regex, JSON schema, EBNF) is also built in.\n\nRequirements are not casual: Linux, GCC 13+, Python 3.12, CUDA 13 (or CANN 9.2 for Ascend).\n\n```\nexport DEEPINFER_ACCELERATOR=nvidia\n./build.sh\npython examples/offline_inference.py\n```\n\nPoint it at local weights with `DEEPINFER_HF_MODEL_DIR=/path/to/DeepSeek-V4.1-Flash-1017`.\n\nThis is **not** a drop-in serving stack. The README says the release is offline inference with the core features for serving at scale, and explicitly leaves out:\n\nThose are exactly the pieces that make or break production serving. DeepSeek ships a companion \"deepseek-recipe\" to reproduce their API's token processing, but you're still assembling the rest yourself.\n\nBenchmarks are Pareto charts (prefill on 4 SM100s, decode on 64) in the repo's images folder. I found no independent head-to-head against vLLM or SGLang yet. Treat any \"X times faster\" tweet as unverified until someone reproduces it. Those two engines have traded the lead for two years, so a single chart proves little.\n\nThe value isn't a vLLM killer. It's that a frontier lab published the serving layer, not just weights. Weights without the engine are half a release, because the engine is where the cost per token lives. The stated goal is to help the community cut compute costs and energy use, and the API-pricing pressure that follows is the real story for anyone paying for tokens.\n\nWatch for two things: someone running independent benchmarks, and the missing disaggregation pieces landing. Until then, it's a seriously impressive reference implementation, not a turnkey replacement.\n\n*Sources: the DeepInfer GitHub repo, The Decoder's coverage, and the Hacker News thread.*", "url": "https://wpnews.pro/news/deepseek-open-sourced-deepinfer-it-s-a-vllm-fork-and-that-s-the-interesting-part", "canonical_source": "https://dev.to/ashraf_chowdury09/deepseek-open-sourced-deepinfer-its-a-vllm-fork-and-thats-the-interesting-part-hak", "published_at": "2026-10-21 09:02:28+00:00", "updated_at": "2026-10-21 09:10:04.432378+00:00", "lang": "en", "topics": ["ai-infrastructure", "large-language-models", "ai-tools", "mlops", "developer-tools"], "entities": ["DeepSeek", "DeepInfer", "vLLM", "SGLang", "TensorRT-LLM", "DeepSeek-V4.1-Flash", "NVIDIA", "Ascend"], "also_reported_by": [], "alternates": {"html": "https://wpnews.pro/news/deepseek-open-sourced-deepinfer-it-s-a-vllm-fork-and-that-s-the-interesting-part", "markdown": "https://wpnews.pro/news/deepseek-open-sourced-deepinfer-it-s-a-vllm-fork-and-that-s-the-interesting-part.md", "text": "https://wpnews.pro/news/deepseek-open-sourced-deepinfer-it-s-a-vllm-fork-and-that-s-the-interesting-part.txt", "jsonld": "https://wpnews.pro/news/deepseek-open-sourced-deepinfer-it-s-a-vllm-fork-and-that-s-the-interesting-part.jsonld"}}