{"slug": "deepinfer-is-open-source-run-deepseeks-inference-engine", "title": "DeepInfer Is Open Source: Run DeepSeek’s Inference Engine", "summary": "DeepSeek open-sourced DeepInfer, the production inference engine behind its API that processes 155 trillion tokens per day, under the Apache 2.0 license. DeepInfer runs only DeepSeek-V4.1-Flash and requires NVIDIA SM100 (Blackwell) GPUs such as B200, GB300 or B300 with CUDA 13, or Huawei Ascend 950 NPUs with CANN 9.2; H100, A100 and consumer GPUs are unsupported. The engine's standout property is batch-invariant inference, producing identical outputs for the same prompt at temperature zero across different batches, but it ships without an HTTP server or KV cache persistence, so users must build their own serving layer.", "body_md": "DeepSeek just open-sourced the actual inference engine running their production API. Not a demo. Not a research prototype. [DeepInfer](https://github.com/deepseek-ai/DeepInfer) is what moves 155 trillion tokens per day through DeepSeek’s clusters — and it is now yours under Apache 2.0. There are real constraints: it only runs DeepSeek-V4.1-Flash, requires NVIDIA Blackwell or Huawei Ascend 950 hardware, and ships without an HTTP server. Here’s what you get, how to set it up, and when you should actually use it over vLLM or SGLang.\n\n## What DeepInfer Actually Is\n\nThis is not a wrapper around vLLM. It is the kernel stack that powered DeepSeek’s API from day one. DeepSeek had previously open-sourced individual components — FlashMLA, DeepGEMM, DeepEP — but DeepInfer bundles them together with the full production logic: speculative decoding via DSpark, confidence-scheduled token verification, and TileLang-based custom GPU kernels.\n\nThe standout property is **batch-invariant inference**. Most inference engines produce slightly different outputs for the same input depending on the batch it lands in. DeepInfer does not. Run the same prompt at temperature zero a thousand times across a thousand different batches and you get the same output every time. For downstream systems that depend on reproducible outputs — evals, deterministic pipelines, regression testing — this matters significantly.\n\nThe companion library [deepseek-recipe](https://github.com/deepseek-ai/deepseek-recipe) handles the codec layer: it converts Messages, Chat Completions, and Responses requests into the token sequences the model actually sees, using the same logic as DeepSeek’s official API. If you care about bit-for-bit parity with DeepSeek’s cloud outputs, deepseek-recipe is what gets you there.\n\n## Who Can Actually Run It\n\nThis is the detail that most coverage glosses over: DeepInfer requires **NVIDIA SM100 (Blackwell) GPUs** — specifically B200, GB300, or B300 cards — with CUDA 13, or **Huawei Ascend 950 NPUs** with CANN 9.2. H100 is not on the list. Neither is A100 or any consumer GPU.\n\nIf you are on Hopper-generation hardware or below, DeepInfer is not for you today. Use vLLM or SGLang instead — both have claimed benchmark parity with DeepInfer on some configurations anyway ([SGLang posted up to 1.04x prefill throughput on GB300/B300 on October 24](https://digg.com/ai/o0sxmv1y)). DeepInfer’s advantage is in the proprietary kernel optimizations and batch invariance, not raw throughput alone.\n\n## Setup: Two Paths\n\nPrerequisites: Linux, GCC 13 or newer, Python 3.12, and the right GPU with the right driver stack.\n\n**Build from source (NVIDIA):**\n\n```\ngit clone https://github.com/deepseek-ai/DeepInfer.git\ncd DeepInfer\npython3.12 -m venv .venv && source .venv/bin/activate\nexport DEEPINFER_ACCELERATOR=nvidia\n./build.sh\n```\n\n**Prebuilt wheel (faster path):** A community-maintained prebuilt is available on [HuggingFace](https://huggingface.co/inferencekey/deepinfer-sm100-cu132) compiled for SM100, CUDA 13.2, Python 3.12:\n\n```\nhuggingface-cli download inferencekey/deepinfer-sm100-cu132 \\\n  --local-dir /workspace/deepinfer-prebuilt\nbash /workspace/deepinfer-prebuilt/install.sh /workspace/di-venv /workspace\n```\n\nOnce installed, run offline inference. The first run auto-downloads the `deepseek-ai/DeepSeek-V4.1-Flash-1017` checkpoint into your HuggingFace cache:\n\n```\npython examples/offline_inference.py\n\n# Or point to a local copy to skip the download\nexport DEEPINFER_HF_MODEL_DIR=/path/to/DeepSeek-V4.1-Flash-1017\npython examples/offline_inference.py\n```\n\n## What It Cannot Do Yet\n\nDeepInfer is an offline inference library. It is not a serving stack. These gaps are significant if you’re planning production deployment:\n\n- **No HTTP server.** There is no built-in OpenAI-compatible endpoint. You will need to build a serving layer on top — FastAPI, a vLLM fork, or wait for the community wrappers to stabilize.\n- **No KV cache persistence.** Each inference call starts cold. Persistent prefix caching across requests is not included.\n- **No load balancing.** Routing, autoscaling, and multi-replica coordination are your problem.\n- **One model only.** DeepSeek-V4.1-Flash-1017 is the only checkpoint officially supported right now.\n\nThe serving layer — KV cache management, prefill/decode disaggregation, replica balancing — is what stayed behind. [DeepSeek flagged these as gaps](https://dev.to/cloudsway/deepseek-opens-deepinfer-what-developers-still-need-to-build-a-serving-stack-gfh) the community will fill or they will add later.\n\n## DeepInfer vs vLLM vs SGLang: When to Use Each\n\nThe decision is straightforward once you frame it correctly:\n\n- **vLLM** : Multi-model serving, OpenAI-compatible API, broad hardware support, production-ready HTTP server. The default choice for anything that isn’t specifically DeepSeek V4.1-Flash on Blackwell hardware.\n- **SGLang** : Prefix-heavy RAG workloads, multi-turn conversations, scenarios with high prefix overlap (60%+). SGLang’s RadixAttention delivers 3-5x prefill latency improvement over vLLM in those conditions.\n- **DeepInfer** : You have Blackwell or Ascend 950 hardware, you are specifically running DeepSeek-V4.1-Flash, and you need batch-invariant deterministic outputs or want to study production-grade MoE inference internals.\n\n## Apache 2.0 and What It Signals\n\nThe Apache 2.0 license is permissive by design: use it commercially, fork it, embed it in a product. The model weights carry their own terms, but the inference framework itself is clean. That is a deliberate signal from DeepSeek about their infrastructure philosophy.\n\nDeepSeek’s open-source trajectory has been systematic: individual kernels first (FlashMLA, DeepGEMM, DeepEP), then the full framework (DeepInfer), with the serving layer presumably next. The pattern suggests a production-grade HTTP serving layer is coming — either from DeepSeek or from the community building on what just dropped.\n\nIf you have the hardware, the framework is [on GitHub now](https://github.com/deepseek-ai/DeepInfer). If you’re on H100 or earlier, bookmark it and check back when the Blackwell install base grows — or when [vLLM or SGLang absorb these kernels](https://blog.premai.io/vllm-vs-sglang-vs-lmdeploy-fastest-llm-inference-engine-in-2026/), which both projects are already moving toward.", "url": "https://wpnews.pro/news/deepinfer-is-open-source-run-deepseeks-inference-engine", "canonical_source": "https://byteiota.com/deepinfer-deepseek-inference-setup/", "published_at": "2026-11-04 19:07:08+00:00", "updated_at": "2026-11-04 19:24:02.409321+00:00", "lang": "en", "topics": ["artificial-intelligence", "large-language-models", "ai-infrastructure", "ai-tools", "ai-chips"], "entities": ["DeepSeek", "DeepInfer", "DeepSeek-V4.1-Flash", "NVIDIA", "Huawei", "vLLM", "SGLang", "deepseek-recipe"], "also_reported_by": [], "alternates": {"html": "https://wpnews.pro/news/deepinfer-is-open-source-run-deepseeks-inference-engine", "markdown": "https://wpnews.pro/news/deepinfer-is-open-source-run-deepseeks-inference-engine.md", "text": "https://wpnews.pro/news/deepinfer-is-open-source-run-deepseeks-inference-engine.txt", "jsonld": "https://wpnews.pro/news/deepinfer-is-open-source-run-deepseeks-inference-engine.jsonld"}}