DeepInfer, DeepSeek's Apache 2.0 Inference Engine, Runs Only DeepSeek-V4.1-Flash

DeepSeek released DeepInfer, the inference framework behind its production services, under the Apache 2.0 license on October 20, supporting only the DeepSeek-V4.1-Flash model. The release ships optimized kernels, attention DP/balanced CP and MoE EP plus EPLB parallelism, DSpark and confidence-scheduled verification, SWA bounded replay, an asynchronous zero-overhead batch scheduler, batch-invariant inference, deterministic sampling and structured outputs. DeepSeek framed the move as sharing its production stack "to help the community lower compute costs, reduce energy use, and make AI more affordable," arriving a week after four labs moved to gate commercial deployment of open-weight models at a $20M revenue line.

Our own logs say a model’s speed and an app’s speed are different numbers. Over 30 days of production calls on our box, qwen3.6:27b decoded at a median 132 tokens per second and delivered 47 to the code that asked for it. Most of that gap is the work around the decode: waiting in a queue, loading the model, reading the prompt. Deciding what runs next, and keeping the GPU busy while it does, is the serving layer’s job. That serving layer is the part DeepSeek just gave away. What shipped On October 20, DeepSeek released DeepInfer https://github.com/deepseek-ai/DeepInfer , the inference framework behind its production services. According to Tech Insider’s writeup https://tech-insider.org/deepseek-deepinfer-open-source-inference-framework-2026/ , it ships under Apache 2.0. That is fully permissive, with no revenue gate attached. The announcement’s own framing, as quoted by Yahoo Tech https://tech.yahoo.com/ai/articles/deepseek-open-sourced-engine-runs-200250940.html : “We’re sharing our production stack to help the community lower compute costs, reduce energy use, and make AI more affordable.” The timing matters. The same piece notes that for a week the industry had been arguing about licenses, with four labs moving to gate commercial deployment of open-weight models at a $20M revenue line. Then DeepSeek moved the pressure to the runtime. Weights get the headlines, but the serving layer is where the cost of inference lives. What’s in the repo The README is short on narrative and long on feature names. Supported model: DeepSeek-V4.1-Flash. One model. The release is scoped as offline inference with “the core features needed for serving at scale.” The feature list, straight from the repo https://github.com/deepseek-ai/DeepInfer : - Optimized kernels, model and sampler runtime - Attention DP / balanced CP, and MoE EP + EPLB for parallelism - DSpark and confidence-scheduled verification - SWA bounded replay - Asynchronous scheduling, described as a zero-overhead batch scheduler - Batch-invariant inference - Deterministic sampling - Structured outputs I can’t tell you what DSpark does from a bullet point, and I won’t guess. Read the feature docs before you build a mental model around it. The rest is more legible. The parallelism line is a cluster story: data-parallel attention, expert parallelism for the mixture-of-experts layers, and a load balancer so no single expert node becomes the hot one. That is a multi-GPU deployment shape. Nothing I read says this targets a single consumer card. The two features I’d read first Batch-invariant inference and deterministic sampling. Most serving stacks have a dirty secret. Run the same prompt twice, once alone and once batched with other requests, and the output can differ. Kernel choices change with batch shape, floating-point math isn’t associative, and the answers drift. Batch invariance means the result doesn’t depend on who else is in the batch. Pair that with deterministic sampling and you get reproducible runs. That matters if you evaluate models for a living. We run QA rails and a judge model in our content pipeline, and a judge that scores the same draft differently depending on queue load is a bad instrument. We’ve already been burned by a judge that failed silently. It returned a safe-default tuple because it was trying to write telemetry to a read-only directory. Silent variance is the same family of bug, just harder to see. The asynchronous scheduler claim is the other one to test. “Zero-overhead” is a strong phrase. It means the GPU shouldn’t sit idle while the CPU plans the next batch. Idle GPU time is the number you can actually measure. That’s why we track raw decode speed against the throughput our application receives. They are never the same number, and the gap is where scheduling, queuing and handoffs live. An engine that claims to close that gap is making a claim you can check on your own hardware. What it means if you run models locally Be honest about scope. This is a production-cluster engine for one model family. If you’re running quantized models on a single card, your stack is probably llama.cpp or Ollama, and DeepInfer won’t replace it this week. But the release still matters to you, for three reasons. It’s a reference implementation. A working, Apache-licensed stack for MoE parallelism and load balancing is something to read even if you never deploy it. The design choices are the value. It shifts the moat. Open weights without an open serving stack meant you got the model but not the economics. Now the economics are readable too. That fits the pattern we laid out in our H1 2026 open-weight roundup https://www.gladlabs.io/posts/open-weight-models-in-h1-2026-qwen-shipped-twice-d-ccf95df2 : the story stopped being one story. It’s software, not silicon. We made a similar argument about DeepSeek’s Ascend tooling https://www.gladlabs.io/posts/deepseek-and-huaweis-ascend-tools-are-a-software-b-9de88982 . The leverage is in the runtime layer. There’s also a planning lesson, the same one from our gpt-oss piece https://www.gladlabs.io/posts/more-than-a-year-after-gpt-oss-openai-has-no-succe-b4cb6894 : build as if the next release isn’t coming. DeepInfer supports one model today. If you depend on it, you depend on DeepSeek’s roadmap. What we’d do first Don’t rip anything out. Clone the repo. Read the feature docs for the items you can’t parse from the bullet list. Then, if you have the hardware, test the claims that are checkable: output stability across batch sizes, and idle GPU time under load. Those two numbers will tell you more than any announcement. The engine is open now, so you can stop taking DeepSeek’s word for it and measure it yourself.