{"slug": "deepseek-open-sources-deepinfer-the-inference-engine-behind-its-production", "title": "DeepSeek open-sources DeepInfer, the inference engine behind its production models", "summary": "DeepSeek open-sourced DeepInfer, the inference framework running its production models, on October 20, 2026 under the Apache-2.0 license. DeepSeek reports that DeepInfer, which powers DeepSeek-V4.1-Flash, sustains 155 trillion tokens per day per cluster with a 96% input cache hit rate, roughly 400 milliseconds average time-to-first-token, and 230 tokens per second generation speed (up to 300 tokens per second for coding tasks), optimized for NVIDIA SM100 GPUs and Huawei Ascend 950 NPUs. The release also includes deepseek-recipe, a token encoding and decoding tool aligned with DeepSeek's official API behavior, aimed at letting enterprises run DeepSeek models on their own hardware at lower compute cost.", "body_md": "Photo: Tara Winstead / Pexels\n\n# DeepSeek open-sources DeepInfer, the inference engine behind its production models\n\nThe Chinese AI lab released its high-performance serving framework under Apache-2.0, alongside a token encoding tool called deepseek-recipe\n\n[DeepSeek](https://cryptobriefing.com/markets/deepseek/) has open-sourced DeepInfer, the high-performance inference framework that runs its own models in production.\n\nThe release landed on October 20, 2026, under the Apache-2.0 license.\n\nThe framework is built for DeepSeek models specifically. It powers the lab’s production-scale deployments, including DeepSeek-V4.1-Flash.\n\n## What DeepInfer actually does\n\nDeepInfer bundles a set of optimizations aimed at making inference faster and cheaper to run. The toolkit includes custom kernels, distributed parallelism, speculative decoding and structured output support.\n\n**Custom kernels** are low-level routines written to squeeze more work out of specific chips.\n\n**Distributed parallelism** splits a model’s workload across many processors at once. The framework includes MoE expert parallelism, which suits mixture-of-experts models. Those models route each query to specialized sub-networks instead of firing up the whole system.\n\n**Speculative decoding** lets a model guess several upcoming tokens ahead of time, then verify them. When the guesses land, output arrives faster.\n\n**Structured output** keeps responses in predictable formats, such as clean JSON.\n\n## The numbers DeepSeek is putting on the table\n\nThe performance figures tied to V4.1-Flash are the eye-catching part of the release.\n\n### AI, tech, and the markets they move—in one daily briefing.\n\nDaily. Free. Join 34,000+ readers across crypto, finance, and policy.\n\nThroughput sits at 155 trillion tokens per day per cluster.\n\nThe input cache hit rate sits at 96%. A hit rate that high suggests most incoming context is being served from memory instead of recomputed from scratch.\n\nAverage time-to-first-token comes in at around 400 milliseconds.\n\nGeneration speed runs at 230 tokens per second, reaching up to 300 tokens per second for coding tasks.\n\nThe framework has been optimized for two hardware families: [NVIDIA](https://cryptobriefing.com/markets/nvidia/) SM100 GPUs and Huawei Ascend 950 NPUs. Its primary target is offline inference, with an emphasis on high throughput while keeping core serving functions intact.\n\n## deepseek-recipe rides along\n\nDeepSeek also released a companion tool called deepseek-recipe.\n\nIt handles token encoding and decoding, the process of turning text into tokens and back again. The tool is designed to match the logic of DeepSeek’s official API.\n\nMismatched tokenization between a self-hosted model and the official service can produce subtle differences in output. By aligning the open tool with its API behavior, DeepSeek makes it easier for teams to run models themselves and get consistent results.\n\n## Part of a longer open-source streak\n\nDeepInfer adds to a series of open-source projects from DeepSeek, which has previously released models and operator libraries.\n\n## What this means\n\nThe most direct beneficiaries are developers and enterprises that want to run DeepSeek models on their own hardware.\n\nThe release is framed as a way for enterprises to cut compute costs and energy usage in large-scale AI deployments.\n\nSupporting both NVIDIA SM100 GPUs and Huawei Ascend 950 NPUs puts a Chinese chip on equal footing with NVIDIA in an official release.\n\nThe framework is tuned for DeepSeek models, so its usefulness beyond that ecosystem is unclear. The focus on offline inference also means teams building latency-sensitive, real-time products may need to evaluate it carefully before switching.\n\n**Disclosure:** This article was edited by Diego Almada Lopez. For more information on how we create and review content, see our\n\n[Editorial Policy](https://cryptobriefing.com/editorial-policy/).", "url": "https://wpnews.pro/news/deepseek-open-sources-deepinfer-the-inference-engine-behind-its-production", "canonical_source": "https://cryptobriefing.com/deepseek-open-sources-deepinfer-inference-framework/", "published_at": "2026-10-20 09:35:11+00:00", "updated_at": "2026-10-20 09:45:58.529529+00:00", "lang": "en", "topics": ["artificial-intelligence", "large-language-models", "ai-infrastructure", "ai-tools", "ai-chips"], "entities": ["DeepSeek", "DeepInfer", "DeepSeek-V4.1-Flash", "deepseek-recipe", "NVIDIA", "Huawei", "NVIDIA SM100", "Huawei Ascend 950"], "also_reported_by": [], "alternates": {"html": "https://wpnews.pro/news/deepseek-open-sources-deepinfer-the-inference-engine-behind-its-production", "markdown": "https://wpnews.pro/news/deepseek-open-sources-deepinfer-the-inference-engine-behind-its-production.md", "text": "https://wpnews.pro/news/deepseek-open-sources-deepinfer-the-inference-engine-behind-its-production.txt", "jsonld": "https://wpnews.pro/news/deepseek-open-sources-deepinfer-the-inference-engine-behind-its-production.jsonld"}}