DeepSeek just open-sourced its inference engine

DeepSeek open-sourced DeepInfer, the inference framework it uses for large-scale production serving, under the Apache License 2.0. The release supports only the DeepSeek-V4.1-Flash model and runs on NVIDIA SM100 GPUs and Huawei Ascend 950 NPUs, with features including FlashMLA, DeepGEMM and DeepEP kernels, MoE expert parallelism with EPLB, DSpark speculative decoding with confidence-scheduled verification, batch-invariant inference and deterministic sampling (both NVIDIA-only), and regex/JSON-schema/EBNF structured outputs. DeepSeek said a complete inference service still requires a KV cache system, prefill/decode disaggregation and load balancing across data-parallel replicas, none of which are included.

DeepInfer is the high-performance inference framework that powers DeepSeek's large-scale production services. This open-source release provides offline inference with the core features needed for serving at scale. Supported models: - DeepSeek-V4.1-Flash Supported features: - Optimized kernels, model and sampler runtime - Parallelism: Attention DP / balanced CP, MoE EP + EPLB - DSpark and confidence-scheduled verification - SWA bounded replay - Asynchronous scheduling zero-overhead batch scheduler - Batch-invariant inference - Deterministic sampling - Structured outputs Feature details and configuration - Optimized kernels: FlashMLA https://github.com/deepseek-ai/FlashMLA , DeepGEMM https://github.com/deepseek-ai/DeepGEMM , DeepEP https://github.com/deepseek-ai/DeepEP , TileKernels https://github.com/deepseek-ai/TileKernels , DeepSelect https://github.com/deepseek-ai/DeepSelect , and custom TileLang kernels https://github.com/deepseek-ai/DeepInfer/blob/main/deepinfer/model executor/layers/tilelang ops . - DSpark and confidence-scheduled verification: accelerate decoding with draft tokens and use confidence estimates to schedule verification. Set DEEPINFER ENABLE MTP VARLEN=1 ; DEEPINFER USE NEXTN DECODE controls the maximum draft length default: 5 . - Decoder SWA bounded replay: reduce decoder prefill work by replaying only a bounded suffix. Set DEEPINFER YOCO PREFILL TRUNCATION MODE=compact and DEEPINFER YOCO PREFILL KEEP TOKENS=140 for the default V4.1 settings; the retained suffix must cover the SWA window plus the maximum backtracking length. - Asynchronous scheduling: overlap CPU batch scheduling with device execution. Enabled by default. - Batch-invariant inference: keep model results identical across different batch compositions. Enable with DEEPINFER BATCH INVARIANT=1 NVIDIA only . - Deterministic sampling: produce identical outputs for identical input tokens and sampling parameters. Set DEEPINFER BATCH INVARIANT=1 DEEPINFER ENABLE MTP VARLEN=0 and pass a fixed seq seed to SamplingParams NVIDIA only . - Structured outputs: constrain generation to a regex, JSON schema, or EBNF grammar. Pass so args={"schema type": ..., "schema": ...} to SamplingParams ; see examples https://github.com/deepseek-ai/DeepInfer/blob/main/examples/ structured output.py . Supported hardware: - NVIDIA SM100 GPUs - Huawei Ascend 950 NPUs Required for a complete inference service not included in this release : - A KV cache system - VE/Prefill/Decode disaggregation most required engine functionality is already implemented - Load balancing across data-parallel replicas for both prefill and decode - Linux - GCC = 13.0 - Python 3.12 - CUDA 13 NVIDIA only - CANN 9.2 Ascend only Before building DeepInfer, please create and activate an empty Python 3.12 environment. Use ascend instead for the Ascend platform Keep it exported in the environment where you run DeepInfer export DEEPINFER ACCELERATOR=nvidia ./build.sh The default checkpoint, deepseek-ai/DeepSeek-V4.1-Flash-1017 https://huggingface.co/deepseek-ai/DeepSeek-V4.1-Flash-1017 , is downloaded into the Hugging Face cache on the first run; set DEEPINFER HF MODEL DIR=/path/to/DeepSeek-V4.1-Flash-1017 to run from a local copy instead. python examples/offline inference.py See examples https://github.com/deepseek-ai/DeepInfer/blob/main/examples/README.md for details. | Prefill | Decode | |---|---| Measured on 4 SM100 GPUs for prefill and 64 for decode — each within one scale-up domain. To reproduce the benchmarks, see run sweep.sh https://github.com/deepseek-ai/DeepInfer/blob/main/examples/sweep/run sweep.sh . See eval/README.md https://github.com/deepseek-ai/DeepInfer/blob/main/eval/README.md . DeepInfer is released under the Apache License 2.0. See LICENSE https://github.com/deepseek-ai/DeepInfer/blob/main/LICENSE for details. DeepInfer was originally built primarily on vLLM https://github.com/vllm-project/vllm , with additional inspiration from SGLang https://github.com/sgl-project/sglang and TensorRT-LLM https://github.com/NVIDIA/TensorRT-LLM . We thank their developers. @misc{deepinfer2026, title = {DeepInfer: High-performance, large-scale inference framework for DeepSeek models}, author = {DeepSeek-AI}, year = {2026}, publisher = {GitHub}, howpublished = {\url{https://github.com/deepseek-ai/DeepInfer}}, }