# DeepSeek Open-Sources DeepInfer, the Engine Serving 155 Trillion Tokens a Day > Source: > Published: 2026-10-21 09:32:14+00:00 DeepSeek has fully open-sourced DeepInfer, the high-performance inference framework that powers its large-scale production services for DeepSeek-V4.1-Flash, along with deepseek-recipe, a production-grade token encoding and decoding component that matches the behavior of the official API. The announcement topped Hacker News within hours, scoring more than 2,100 points, and is being called the first time a frontier lab has handed over its entire production serving engine rather than just isolated components. What was released DeepInfer, published under the Apache 2.0 license on GitHub, bundles optimized kernels including FlashMLA, DeepGEMM, DeepEP and TileKernels, attention and mixture-of-experts parallelism, DSpark speculative decoding with confidence-scheduled verification, asynchronous scheduling, batch-invariant inference, deterministic sampling and structured outputs. The framework was built primarily on a deeply modified fork of vLLM, with inspiration from SGLang and TensorRT-LLM. The companion deepseek-recipe library handles conversion between mainstream LLM API protocols and model tokens, with processing logic fully aligned to DeepSeek's official API, so third-party deployments can reproduce the hosted experience as closely as possible. DeepSeek says inference code for future models will land in the same repository. Production-proven performance The numbers DeepSeek published describe one of the most efficient serving stacks in the industry: a single cluster handles 155 trillion tokens per day, with off-peak revenue of about 38 dollars per GPU per hour. Around 96 percent of inputs hit the prefix cache, lifting per-GPU prefill throughput from 60,000 to 1.5 million tokens per second — a 25-fold difference. First-token latency is about 400 milliseconds with generation at 230 tokens per second. Two techniques stand out. DSpark speculative decoding has a small draft model propose five tokens at a time while a confidence head predicts which the main model will accept, cutting verification work and adding 4 to 7 percent throughput at production loads. SWA Bounded Replay fixes cache-miss recomputation to the last 128 tokens of a prompt, nearly halving total prefill compute and making long-context costs largely independent of context length. MoE expert replicas are dynamically reshuffled every 10,000 steps, keeping the busiest GPU within 9 percent of average load. Dual hardware support and what is missing DeepInfer runs on both NVIDIA SM100 (Blackwell) data center GPUs and Huawei Ascend 950 NPUs, making it one of the very few production-grade inference frameworks spanning both compute ecosystems — a significant signal for Chinese firms reducing reliance on NVIDIA while still serving global customers. Requirements are Linux, Python 3.12, GCC 13 or later, and either CUDA 13 or CANN 9.2. The release is not a complete serving stack: KV cache management and cross-machine load balancing remain closed, as they are tightly coupled to DeepSeek's internal infrastructure. Currently only offline inference for DeepSeek-V4.1-Flash is supported. By open-sourcing its production engine, DeepSeek puts further pressure on Western providers on pricing and gives developers a practical path to dramatically lower serving costs. Sources: InfoQ, SiliconANGLE, TechTarget, The Next Web, dev.to, threerouter