Photo: Tara Winstead / Pexels
The Chinese AI lab released its high-performance serving framework under Apache-2.0, alongside a token encoding tool called deepseek-recipe
DeepSeek has open-sourced DeepInfer, the high-performance inference framework that runs its own models in production.
The release landed on October 20, 2026, under the Apache-2.0 license.
The framework is built for DeepSeek models specifically. It powers the lab’s production-scale deployments, including DeepSeek-V4.1-Flash.
What DeepInfer actually does #
DeepInfer bundles a set of optimizations aimed at making inference faster and cheaper to run. The toolkit includes custom kernels, distributed parallelism, speculative decoding and structured output support.
Custom kernels are low-level routines written to squeeze more work out of specific chips.
Distributed parallelism splits a model’s workload across many processors at once. The framework includes MoE expert parallelism, which suits mixture-of-experts models. Those models route each query to specialized sub-networks instead of firing up the whole system.
Speculative decoding lets a model guess several upcoming tokens ahead of time, then verify them. When the guesses land, output arrives faster.
Structured output keeps responses in predictable formats, such as clean JSON.
The numbers DeepSeek is putting on the table #
The performance figures tied to V4.1-Flash are the eye-catching part of the release.
AI, tech, and the markets they move—in one daily briefing.
Daily. Free. Join 34,000+ readers across crypto, finance, and policy.
Throughput sits at 155 trillion tokens per day per cluster.
The input cache hit rate sits at 96%. A hit rate that high suggests most incoming context is being served from memory instead of recomputed from scratch.
Average time-to-first-token comes in at around 400 milliseconds.
Generation speed runs at 230 tokens per second, reaching up to 300 tokens per second for coding tasks.
The framework has been optimized for two hardware families: NVIDIA SM100 GPUs and Huawei Ascend 950 NPUs. Its primary target is offline inference, with an emphasis on high throughput while keeping core serving functions intact.
deepseek-recipe rides along #
DeepSeek also released a companion tool called deepseek-recipe.
It handles token encoding and decoding, the process of turning text into tokens and back again. The tool is designed to match the logic of DeepSeek’s official API.
Mismatched tokenization between a self-hosted model and the official service can produce subtle differences in output. By aligning the open tool with its API behavior, DeepSeek makes it easier for teams to run models themselves and get consistent results.
Part of a longer open-source streak #
DeepInfer adds to a series of open-source projects from DeepSeek, which has previously released models and operator libraries.
What this means #
The most direct beneficiaries are developers and enterprises that want to run DeepSeek models on their own hardware.
The release is framed as a way for enterprises to cut compute costs and energy usage in large-scale AI deployments.
Supporting both NVIDIA SM100 GPUs and Huawei Ascend 950 NPUs puts a Chinese chip on equal footing with NVIDIA in an official release.
The framework is tuned for DeepSeek models, so its usefulness beyond that ecosystem is unclear. The focus on offline inference also means teams building latency-sensitive, real-time products may need to evaluate it carefully before switching.
Disclosure: This article was edited by Diego Almada Lopez. For more information on how we create and review content, see our