Back to blog October 22, 2026 · 8 min read
We compared DeepSeek's DeepInfer with vLLM v0.32.0 through GitHits, tracing scheduling, hardware backends and speculative decoding to source.
DeepSeek open-sourced DeepInfer this week under Apache 2.0. The README calls it the inference framework behind its production services, and credits vLLM, with inspiration from SGLang and TensorRT-LLM. Twenty-three files still open with # Derived from vLLM; modified by DeepSeek. The interesting question is how far the fork has actually drifted.
At 774ce244, three commits in and not yet tagged, DeepInfer is an offline engine for one model. vLLM v0.32.0 is a general server, and accelerators beyond its five built-in platforms load as plugins. We read vllm-ascend at v0.30.0rc2, the newest tag on that plugin the day we looked. On confidence-based speculative verification, both maximize expected accepted tokens per unit of step cost. We read the source through GitHits and did not run either engine. The performance numbers below are the ones the repositories publish.
| Question | DeepInfer (774ce244) | vLLM (v0.32.0) |
|---|---|---|
| Scope | One supported model, DeepSeek-V4.1-Flash, on NVIDIA SM100 and Ascend 950 | A model registry with many architectures, including DeepSeek V2 through V4.1 |
| How you call it | LLM.generate() in Python; the only HTTP route isPOST /profile | OpenAI-compatible HTTP routes and gRPC, plus LLM.generate() |
| Scheduler | C++ header; data-parallel ranks vote on prefill or decode each step | Python; no prefill or decode phase, one token budget per step, chunked prefill |
| KV cache blocks | C++ reference-counted free list; admission answers OK, LATER or NEVER | Python block pool with prefix caching on by default |
| Parallelism | Expert, data and context parallel plus an “engram” group; no tensor or pipeline parallel | Tensor, pipeline, data, prefill-context and decode-context parallel, with an expert-parallel switch |
| Expert load balancing | Greedy two-stage replica placement in Python | Default policy adapted from DeepSeek’s EPLB |
| Adaptive speculative verification | C++, scheduler side, opt-in | Python, GPU worker side, opt-in, DSpark only |
| CPU-GPU sync guard | On by default; syncs in forward and sample raise | Off by default; warn orerror after warmup |
| Ascend | In-tree PlatformNPU | Separate vllm-ascend plugin repository |
One way in, or a server #
DeepInfer is a script. generate() adds the requests and calls llm_engine.step() until nothing is left. The only HTTP route is POST /profile, a profiler trigger, and the batch is DeepSeek-V4.1-Flash. vLLM keeps the offline LLM.generate() and also speaks /v1/chat/completions, /v1/completions, /v1/models, gRPC, and others. Both front ends reach EngineCore through an EngineCoreClient, from llm_engine.py and async_llm.py. A batch can stop at DeepInfer. A client on the network needs vLLM.
DeepInfer’s scheduler is a C++ header. Under data parallelism each rank is a replica with its own batch. The ranks allreduce until they agree the next step is prefill, processing a prompt, or decode, producing the next token, and every rank runs that kind of step. The replicas stay in lockstep. A prompt and a generation do not share it.
vLLM’s scheduler has no decoding phase and no prefill phase. Each request remembers how many tokens are already computed, and each step hands out a token budget so requests can catch up. Long prompts are split into chunks. enable_chunked_prefill defaults to true on the config class, though the docstring says the engine arguments normally set it. One step on a shared GPU can hold a prompt and someone else’s next token.
When a vLLM rank has nothing ready and the others are still running, DPEngineCoreProc runs a dummy pass. With enable_balance_scheduling, patch_balance_schedule.py patches vLLM’s Scheduler, because prefill and decode in one pass “can impact the overall system throughput and performance in some scenarios”.
DeepInfer’s block manager is a C++ reference-counted free list, and the answer is OK, LATER, or NEVER. vLLM’s BlockPool keeps free blocks in eviction order so a cached prefix can be found by hash, and prefix caching is on by default.
NVIDIA and Ascend #
NVIDIA and Ascend both live in DeepInfer’s tree, and the model code does not branch on which one it got. PlatformGPU selects the cuda kernels, nccl, and int32 FP8 scales. PlatformNPU selects asc and hccl, with int16 scales. A 32-line dispatcher loads the kernel set, and build_vendor.sh builds a different library per vendor under the same four import names. flash_mla or deep_gemm is the same import on NVIDIA SM100 and on Ascend 950.
Five platforms are built in: TPU, CUDA, ROCm, XPU, and CPU. vLLM also loads any installed package registered under vllm.platform_plugins. vllm-ascend joins as ascend = vllm_ascend:register, which returns NPUPlatform. The platform sets device_name to "npu" and uses PrivateUse1, the PyTorch dispatch key reserved for devices that ship out of tree. It carries its own models/deepseek_v41/ and a patch/ package that edits vLLM classes at startup. The plugin is versioned alongside vLLM, and the tags we found were v0.30.0rc2 against v0.32.0. Choosing Ascend on this side means following both repositories.
vLLM already registers DeepseekV4ForCausalLM and DeepseekV41ForCausalLM, with per-vendor code under vllm/models/deepseek_v41/ for NVIDIA and AMD. DeepInfer’s list is the one model. “Engram” shows up in DeepInfer’s parallel state with no explanation beside it. The clearest note we found is vLLM’s, “n-gram hash lookups gated into the hyper-connection stream”, ported from DeepSeek’s reference code.
ParallelConfig is the wider menu: tensor, pipeline, prefill-context, and data parallel sizes, enable_expert_parallel, enable_eplb, and decode-context parallelism. We searched DeepInfer for tensor or pipeline parallel names and found none. DeepSeek’s deployment Q&A says production does not use tensor parallelism. The groups here are expert, data, and context, plus engram. A tensor or pipeline setting from a vLLM config has nothing to map onto.
Expert load balancing, copying busy experts onto more GPUs, is where the credit runs the other way. vLLM’s default policy says it was adapted from DeepSeek’s EPLB. DeepInfer has its own greedy two-stage replication and placement.
KV formats overlap and are spelled differently. vLLM’s cache_dtype accepts fp8_e4m3, fp8_e5m2, fp8_ds_mla, and nvfp4_ds_mla, and quantization includes deepseek_v4_fp8. DeepInfer declares five cache formats for its one model, with block scales stored per 16, 32 or 128 elements.
Speculative verification #
Speculative decoding drafts several tokens with a cheap model, then checks them in one forward pass of the full model. DeepInfer sizes that check in C++ beside the scheduler. vLLM sizes it in Python on the GPU worker. Both take that size from the draft model’s own confidence.
A ConfidenceHead predicts acceptance per draft token. MTPVarlen sorts pending drafts by that rate and keeps the batch size with the most expected tokens per predicted step time. Step time is a median over a 5,000-step window, so the cost is whatever this process has actually been paying. The path is opt-in, behind DEEPINFER_ENABLE_MTP_VARLEN.
AdaptiveVerificationManager scores each draft slot by the running product of confidences, prices the step from cost tables seeded by profiling runs at startup, and takes the argmax of expected accepted tokens over that cost. It is off by default, and only for method="dspark".
DeepSeek’s Q&A, which we did not reproduce, says this cuts draft tokens verified per step by about 30%, from 5.00 to 3.2–4.4, with accepted length roughly unchanged, and estimates +4–7% throughput at about 60 concurrent sequences per rank from EP64/DP64 benchmark data. The table there shows a 1.5% loss at 16 sequences per rank. Those figures are DeepSeek’s. We found no comparable figure in vLLM’s source.
Both use torch’s sync debug mode, which is process-wide. DeepInfer wraps forward and sample in forbid_cuda_sync() and raises unless DEEPINFER_DISABLE_SYNC_GUARD is set. vLLM’s gpu_sync_debug catches those syncs and non-blocking copies through pageable memory, arms only after warmup, and stays off unless VLLM_GPU_SYNC_CHECK is warn or error.
What the credit line still matches #
The headers never name a vLLM version. We compared each of the 23 files with its counterpart at 16 tags from v0.2.0 to v0.32.0 and counted how many of DeepInfer’s non-comment lines still appear verbatim. It is a rough measure. Sixteen of the 23 are closest to a release from v0.2.0 to v0.5.0, and the median overlap is 19%. The nearest are the shared-memory broadcast queue, 73% and best against v0.6.6 and v0.7.3, and model_executor/models/__init__.py at 70%. Most of the rest kept the filename and rewrote the body. The buffer-moving bits still look like vLLM. The engine around them is DeepInfer’s.
vLLM v0.3.3’s CacheEngine is 178 lines and swaps blocks to CPU. DeepInfer’s is 978 lines, does not swap, and adds a KVCacheManager plus sliding-window state. sequence.py replaces vLLM’s sequence-group classes with MTPPredictInfo and SampledInfo. sampling_params.py removes beam search. Two of the 23 have no counterpart at v0.32.0, because vLLM’s V0 worker/ directory is gone. The percentages are how you tell a small patch from a file that grew up.
Use the GitHits API #
Each link above is a file and a line at the revision we read, the same jump the API gives an agent. The API overview maps those calls. Create a key in the token settings of your GitHits account and send it as a bearer token. The authentication guide covers storing that key for CI, and the quickstart is the first request.
If you’re a software factory interested in using our API, we’d love for you to book a call with us to discuss what it would look like.