DeepSeek has open-sourced DeepInfer, the engine behind its own production service. We put it head to head with vLLM on a single 8×B300 node running DeepSeek-V4.1-Flash-1017, and pushed both to the limit on output tokens and on prefill.
TL;DR: on one node vLLM is 2.0× faster on decode (1K → 1K), 4.3× at 8K → 1K, and 9× on prefill. It peaks at 658K prefill tokens/s and 55–65K output tokens/s.
Both run the same model with mostly the same kernels (FlashMLA, DeepGEMM, DeepEP) and scale it the same way: attention replicated per GPU, experts sharded across GPUs. The difference is scope.
| vLLM | DeepInfer | |
|---|---|---|
| Models | Hundreds | DeepSeek-V4.1-Flash only |
| Hardware | NVIDIA, AMD, TPU… | NVIDIA SM100, Huawei Ascend 950 |
| Interface | OpenAI-compatible server | Python offline API ( LLM.generate ), no server |
| Install | Docker / pip | Build from source (CUDA 13.1+) |
| Highlights | Prefix caching, structured outputs, LoRA, disaggregated prefill | DSpark speculative decoding, batch-invariant inference, deterministic sampling |
| Built for | Any scale | 4 GPUs per prefill group, 64 per decode group |
vllm bench serve, random-token prompts, fixed output length; DeepInfer got the same prompts in-process
Random prompts are the worst case for speculative decoding (19% draft acceptance at 1K, 12% at 8K), so every decode number here is a floor.
The model is MoE: 384 routed experts + 1 shared per layer, 6 per token. vLLM's recipe runs it as DEP8: each GPU has its own attention and KV cache and holds 48 experts; tokens travel to their experts over NVSwitch every MoE layer.
Memory is not the bottleneck: weights take 46 GiB per GPU and the KV cache gets 160 GiB (room for ~52M tokens per rank). The real limit is 256 running sequences per rank, 2,048 per node.
Prefill (one output token per request): 658K tokens/s with 8K prompts, about 82K per GPU.
Decode with 2,048 concurrent requests: 65.0K output tokens/s with short prompts, 55.0K at 1K → 1K (29 ms per token).
| In → Out | Concurrent | Output tok/s | Total tok/s | TPOT |
|---|---|---|---|---|
| 128 → 1K | 2,048 | 64,990 | 73,114 | 24.8 ms |
| 1K → 1K | 2,048 | 54,985 | 109,969 | 28.8 ms |
| 8K → 1K | 2,048 | 29,500 | 265,496 | 52.8 ms |
| 2K → 128 | 2,048 | 19,753 | 335,801 | 74.4 ms |
One gotcha: our first run made only 37.2K tok/s at 1K → 1K because CUDA graphs were captured only up to 1,536 tokens. Under load, steps mix decode with prompt chunks and every DP rank pads to the largest, so those steps ran without graphs. Capturing graphs up to 8,192 tokens brought it to 55.0K.
Getting it running took longer than benchmarking it: source build with NVCC 13.1+, and it needs an RDMA device even on a single node (NCCL GPU-initiated networking for Engram rows).
| In → Out | Concurrent | DeepInfer | vLLM | vLLM ÷ DeepInfer |
|---|---|---|---|---|
| 1K → 1K | 2,048 | 27,906 | 54,985 | 2.0× |
| 1K → 1K | 4,096 | 26,294 | 55,506 | 2.1× |
| 8K → 1K | 2,048 | 6,933 | 29,500 | 4.3× |
| 32K → 1 (prefill) | 512 | 64,847 | 589,924 | 9.1× |
DeepInfer's own benchmark agrees (66K prefill tok/s per node), so it's not our harness.
Why the gap? DeepSeek benchmarks DeepInfer with prefill on 4 GPUs and decode on 64 GPUs, each in one NVLink domain. That's a rack, not a server, with prefill and decode on separate machines. The pieces that hold such a deployment together (KV cache system, P/D disaggregation, load balancing) aren't in the open-source release. On one node DeepInfer has to do everything at once, which is exactly what vLLM's recipe is built for. So read this as "DeepInfer on one node", not a verdict on its kernels.
For most teams with a node or two, the answer is vLLM. DeepInfer is worth following as the reference for how DeepSeek runs its own model, but on a single node it isn't the faster engine.
*Originally published at [warmctl.io](https://warmctl.io/blog/deepseek-b300-vllm-vs-deepinfer/), with the full setup, startup-time breakdown and raw numbers.*