{"slug": "deepseek-v4-1-flash-on-8-b300-vllm-vs-deepinfer-600k-tokens-s", "title": "DeepSeek V4.1 Flash on 8 B300: vLLM vs DeepInfer, 600K+ tokens/s", "summary": "A developer benchmarked DeepSeek's newly open-sourced DeepInfer inference engine against vLLM on a single 8×B300 node running DeepSeek-V4.1-Flash-1017, finding vLLM 2.0× faster on decode (1K→1K), 4.3× at 8K→1K, and 9× on prefill, peaking at 658K prefill tokens/s and 55–65K output tokens/s. The gap stems from scope: DeepSeek tunes DeepInfer for a rack-scale deployment with prefill on 4 GPUs and decode on 64, and the KV cache, P/D disaggregation and load-balancing components aren't in the open-source release. The developer concludes that for teams with a node or two, vLLM is the faster engine on a single node.", "body_md": "DeepSeek has open-sourced DeepInfer, the engine behind its own production service. We put it head to head with vLLM on a single 8×B300 node running DeepSeek-V4.1-Flash-1017, and pushed both to the limit on output tokens and on prefill.\n\n**TL;DR:** on one node vLLM is **2.0× faster on decode** (1K → 1K), **4.3× at 8K → 1K**, and **9× on prefill**. It peaks at **658K prefill tokens/s** and **55–65K output tokens/s**.\n\nBoth run the same model with mostly the same kernels (FlashMLA, DeepGEMM, DeepEP) and scale it the same way: attention replicated per GPU, experts sharded across GPUs. The difference is scope.\n\n|  | vLLM | DeepInfer | \n|---|---|---|\n| Models | Hundreds | DeepSeek-V4.1-Flash only | \n| Hardware | NVIDIA, AMD, TPU… | NVIDIA SM100, Huawei Ascend 950 | \n| Interface | OpenAI-compatible server | Python offline API ( `LLM.generate` ), no server | \n| Install | Docker / pip | Build from source (CUDA 13.1+) | \n| Highlights | Prefix caching, structured outputs, LoRA, disaggregated prefill | DSpark speculative decoding, batch-invariant inference, deterministic sampling | \n| Built for | Any scale | 4 GPUs per prefill group, 64 per decode group | \n\n`vllm bench serve`, random-token prompts, fixed output length; DeepInfer got the same prompts in-process\nRandom prompts are the worst case for speculative decoding (19% draft acceptance at 1K, 12% at 8K), so every decode number here is a floor.\n\nThe model is MoE: 384 routed experts + 1 shared per layer, 6 per token. vLLM's recipe runs it as DEP8: each GPU has its own attention and KV cache and holds 48 experts; tokens travel to their experts over NVSwitch every MoE layer.\n\nMemory is not the bottleneck: weights take 46 GiB per GPU and the KV cache gets 160 GiB (room for ~52M tokens per rank). The real limit is 256 running sequences per rank, 2,048 per node.\n\n**Prefill** (one output token per request): 658K tokens/s with 8K prompts, about 82K per GPU.\n\n**Decode** with 2,048 concurrent requests: 65.0K output tokens/s with short prompts, 55.0K at 1K → 1K (29 ms per token).\n\n| In → Out | Concurrent | Output tok/s | Total tok/s | TPOT | \n|---|---|---|---|---|\n| 128 → 1K | 2,048 | 64,990 | 73,114 | 24.8 ms | \n| 1K → 1K | 2,048 | 54,985 | 109,969 | 28.8 ms | \n| 8K → 1K | 2,048 | 29,500 | 265,496 | 52.8 ms | \n| 2K → 128 | 2,048 | 19,753 | 335,801 | 74.4 ms | \n\nOne gotcha: our first run made only 37.2K tok/s at 1K → 1K because CUDA graphs were captured only up to 1,536 tokens. Under load, steps mix decode with prompt chunks and every DP rank pads to the largest, so those steps ran without graphs. Capturing graphs up to 8,192 tokens brought it to 55.0K.\n\nGetting it running took longer than benchmarking it: source build with NVCC 13.1+, and it needs an RDMA device even on a single node (NCCL GPU-initiated networking for Engram rows).\n\n| In → Out | Concurrent | DeepInfer | vLLM | vLLM ÷ DeepInfer | \n|---|---|---|---|---|\n| 1K → 1K | 2,048 | 27,906 | 54,985 | 2.0× | \n| 1K → 1K | 4,096 | 26,294 | 55,506 | 2.1× | \n| 8K → 1K | 2,048 | 6,933 | 29,500 | 4.3× | \n| 32K → 1 (prefill) | 512 | 64,847 | 589,924 | 9.1× | \n\nDeepInfer's own benchmark agrees (66K prefill tok/s per node), so it's not our harness.\n\n**Why the gap?** DeepSeek benchmarks DeepInfer with prefill on 4 GPUs and decode on 64 GPUs, each in one NVLink domain. That's a rack, not a server, with prefill and decode on separate machines. The pieces that hold such a deployment together (KV cache system, P/D disaggregation, load balancing) aren't in the open-source release. On one node DeepInfer has to do everything at once, which is exactly what vLLM's recipe is built for. So read this as \"DeepInfer on one node\", not a verdict on its kernels.\n\nFor most teams with a node or two, the answer is vLLM. DeepInfer is worth following as the reference for how DeepSeek runs its own model, but on a single node it isn't the faster engine.\n\n*Originally published at [warmctl.io](https://warmctl.io/blog/deepseek-b300-vllm-vs-deepinfer/), with the full setup, startup-time breakdown and raw numbers.*", "url": "https://wpnews.pro/news/deepseek-v4-1-flash-on-8-b300-vllm-vs-deepinfer-600k-tokens-s", "canonical_source": "https://dev.to/kbespalov/deepseek-v41-flash-on-8xb300-vllm-vs-deepinfer-600k-tokenss-3jl9", "published_at": "2026-10-21 15:04:21+00:00", "updated_at": "2026-10-21 15:10:09.234948+00:00", "lang": "en", "topics": ["ai-infrastructure", "large-language-models", "ai-tools", "mlops"], "entities": ["DeepSeek", "vLLM", "DeepInfer", "NVIDIA", "DeepSeek-V4.1-Flash-1017", "B300", "Huawei"], "also_reported_by": [], "alternates": {"html": "https://wpnews.pro/news/deepseek-v4-1-flash-on-8-b300-vllm-vs-deepinfer-600k-tokens-s", "markdown": "https://wpnews.pro/news/deepseek-v4-1-flash-on-8-b300-vllm-vs-deepinfer-600k-tokens-s.md", "text": "https://wpnews.pro/news/deepseek-v4-1-flash-on-8-b300-vllm-vs-deepinfer-600k-tokens-s.txt", "jsonld": "https://wpnews.pro/news/deepseek-v4-1-flash-on-8-b300-vllm-vs-deepinfer-600k-tokens-s.jsonld"}}