DeepInfer vs vLLM: What the Source Shows

DeepSeek open-sourced DeepInfer, the inference framework behind its production services, under Apache 2.0 this week, and a source-level comparison by GitHits against vLLM v0.32.0 finds the fork has drifted substantially from its vLLM origins. At commit 774ce244, three commits in and untagged, DeepInfer is an offline engine supporting only DeepSeek-V4.1-Flash on NVIDIA SM100 and Ascend 950, with a C++ scheduler and KV cache, while vLLM v0.32.0 remains a general server with OpenAI-compatible HTTP and gRPC routes and a multi-architecture model registry. Twenty-three DeepInfer files still carry the header "Derived from vLLM; modified by DeepSeek," and the comparison was read from source via GitHits without running either engine, so all performance figures cited are those the repositories publish.

Back to blog https://githits.com/blog/ October 22, 2026 · 8 min read DeepInfer vs vLLM: What the Source Shows We compared DeepSeek's DeepInfer with vLLM v0.32.0 through GitHits, tracing scheduling, hardware backends and speculative decoding to source. DeepSeek open-sourced DeepInfer https://github.com/deepseek-ai/DeepInfer/blob/774ce244b87cc0eb8d4b94ab9f5de2ab22049f01/README.md L3 this week under Apache 2.0. The README calls it the inference framework behind its production services, and credits vLLM, with inspiration from SGLang and TensorRT-LLM https://github.com/deepseek-ai/DeepInfer/blob/774ce244b87cc0eb8d4b94ab9f5de2ab22049f01/README.md L95 . Twenty-three files still open with Derived from vLLM; modified by DeepSeek. The interesting question is how far the fork has actually drifted. At 774ce244 https://github.com/deepseek-ai/DeepInfer/blob/774ce244b87cc0eb8d4b94ab9f5de2ab22049f01 , three commits in and not yet tagged, DeepInfer is an offline engine for one model. vLLM v0.32.0 https://github.com/vllm-project/vllm/blob/5df63a7203cdb9328d824a27b682e77c7b9928be is a general server, and accelerators beyond its five built-in platforms load as plugins. We read vllm-ascend https://github.com/vllm-project/vllm-ascend/blob/8fde6d4e335fe300e2b949aab0e31659cce0811d at v0.30.0rc2 , the newest tag on that plugin the day we looked. On confidence-based speculative verification, both maximize expected accepted tokens per unit of step cost. We read the source through GitHits and did not run either engine. The performance numbers below are the ones the repositories publish. | Question | DeepInfer 774ce244 | vLLM v0.32.0 | |---|---|---| | Scope | One supported model, DeepSeek-V4.1-Flash, on NVIDIA SM100 and Ascend 950 | A model registry with many architectures, including DeepSeek V2 through V4.1 | | How you call it | LLM.generate in Python; the only HTTP route is POST /profile | OpenAI-compatible HTTP routes and gRPC, plus LLM.generate | | Scheduler | C++ header; data-parallel ranks vote on prefill or decode each step | Python; no prefill or decode phase, one token budget per step, chunked prefill | | KV cache blocks | C++ reference-counted free list; admission answers OK, LATER or NEVER | Python block pool with prefix caching on by default | | Parallelism | Expert, data and context parallel plus an “engram” group; no tensor or pipeline parallel | Tensor, pipeline, data, prefill-context and decode-context parallel, with an expert-parallel switch | | Expert load balancing | Greedy two-stage replica placement in Python | Default policy adapted from DeepSeek’s EPLB | | Adaptive speculative verification | C++, scheduler side, opt-in | Python, GPU worker side, opt-in, DSpark only | | CPU-GPU sync guard | On by default; syncs in forward and sample raise | Off by default; warn or error after warmup | | Ascend | In-tree PlatformNPU | Separate vllm-ascend plugin repository | One way in, or a server DeepInfer is a script. generate https://github.com/deepseek-ai/DeepInfer/blob/774ce244b87cc0eb8d4b94ab9f5de2ab22049f01/deepinfer/entrypoints/llm.py L90 adds the requests and calls llm engine.step until nothing is left https://github.com/deepseek-ai/DeepInfer/blob/774ce244b87cc0eb8d4b94ab9f5de2ab22049f01/deepinfer/entrypoints/llm.py L176-L178 . The only HTTP route is POST /profile https://github.com/deepseek-ai/DeepInfer/blob/774ce244b87cc0eb8d4b94ab9f5de2ab22049f01/deepinfer/engine/profile server.py L1-L12 , a profiler trigger, and the batch is DeepSeek-V4.1-Flash. vLLM keeps the offline LLM.generate https://github.com/vllm-project/vllm/blob/5df63a7203cdb9328d824a27b682e77c7b9928be/vllm/entrypoints/llm.py L418-L424 and also speaks /v1/chat/completions https://github.com/vllm-project/vllm/blob/5df63a7203cdb9328d824a27b682e77c7b9928be/vllm/entrypoints/openai/chat completion/api router.py L41-L42 , /v1/completions https://github.com/vllm-project/vllm/blob/5df63a7203cdb9328d824a27b682e77c7b9928be/vllm/entrypoints/openai/completion/api router.py L35-L36 , /v1/models https://github.com/vllm-project/vllm/blob/5df63a7203cdb9328d824a27b682e77c7b9928be/vllm/entrypoints/openai/models/api router.py L20 , gRPC, and others. Both front ends reach EngineCore through an EngineCoreClient , from llm engine.py https://github.com/vllm-project/vllm/blob/5df63a7203cdb9328d824a27b682e77c7b9928be/vllm/v1/engine/llm engine.py L116 and async llm.py https://github.com/vllm-project/vllm/blob/5df63a7203cdb9328d824a27b682e77c7b9928be/vllm/v1/engine/async llm.py L184 . A batch can stop at DeepInfer. A client on the network needs vLLM. DeepInfer’s scheduler is a C++ header https://github.com/deepseek-ai/DeepInfer/blob/774ce244b87cc0eb8d4b94ab9f5de2ab22049f01/csrc/engine/scheduler.hpp L308-L318 . Under data parallelism each rank is a replica with its own batch. The ranks allreduce until they agree the next step is prefill, processing a prompt, or decode, producing the next token, and every rank runs that kind of step. The replicas stay in lockstep. A prompt and a generation do not share it. vLLM’s scheduler has no decoding phase and no prefill phase https://github.com/vllm-project/vllm/blob/5df63a7203cdb9328d824a27b682e77c7b9928be/vllm/v1/core/sched/scheduler.py L594-L600 . Each request remembers how many tokens are already computed, and each step hands out a token budget so requests can catch up. Long prompts are split into chunks. enable chunked prefill defaults to true on the config class https://github.com/vllm-project/vllm/blob/5df63a7203cdb9328d824a27b682e77c7b9928be/vllm/config/scheduler.py L134-L140 , though the docstring says the engine arguments normally set it. One step on a shared GPU can hold a prompt and someone else’s next token. When a vLLM rank has nothing ready and the others are still running, DPEngineCoreProc runs a dummy pass https://github.com/vllm-project/vllm/blob/5df63a7203cdb9328d824a27b682e77c7b9928be/vllm/v1/engine/core.py L2353-L2363 . With enable balance scheduling , patch balance schedule.py https://github.com/vllm-project/vllm-ascend/blob/8fde6d4e335fe300e2b949aab0e31659cce0811d/vllm ascend/patch/ init .py L35-L50 patches vLLM’s Scheduler , because prefill and decode in one pass “can impact the overall system throughput and performance in some scenarios”. DeepInfer’s block manager https://github.com/deepseek-ai/DeepInfer/blob/774ce244b87cc0eb8d4b94ab9f5de2ab22049f01/csrc/engine/block manager.hpp L14-L18 is a C++ reference-counted free list, and the answer is OK , LATER , or NEVER . vLLM’s BlockPool https://github.com/vllm-project/vllm/blob/5df63a7203cdb9328d824a27b682e77c7b9928be/vllm/v1/core/block pool.py L136-L146 keeps free blocks in eviction order so a cached prefix can be found by hash, and prefix caching is on by default https://github.com/vllm-project/vllm/blob/5df63a7203cdb9328d824a27b682e77c7b9928be/vllm/config/cache.py L141 . NVIDIA and Ascend NVIDIA and Ascend both live in DeepInfer’s tree, and the model code does not branch on which one it got. PlatformGPU https://github.com/deepseek-ai/DeepInfer/blob/774ce244b87cc0eb8d4b94ab9f5de2ab22049f01/deepinfer/platform.py L55-L63 selects the cuda kernels, nccl , and int32 FP8 scales. PlatformNPU https://github.com/deepseek-ai/DeepInfer/blob/774ce244b87cc0eb8d4b94ab9f5de2ab22049f01/deepinfer/platform.py L86-L90 selects asc and hccl , with int16 scales https://github.com/deepseek-ai/DeepInfer/blob/774ce244b87cc0eb8d4b94ab9f5de2ab22049f01/deepinfer/platform.py L138 . A 32-line dispatcher https://github.com/deepseek-ai/DeepInfer/blob/774ce244b87cc0eb8d4b94ab9f5de2ab22049f01/deepinfer/model executor/layers/tilelang ops/ops.py L15-L32 loads the kernel set, and build vendor.sh https://github.com/deepseek-ai/DeepInfer/blob/774ce244b87cc0eb8d4b94ab9f5de2ab22049f01/scripts/build vendor.sh L7-L22 builds a different library per vendor under the same four import names. flash mla or deep gemm is the same import on NVIDIA SM100 and on Ascend 950. Five platforms are built in https://github.com/vllm-project/vllm/blob/5df63a7203cdb9328d824a27b682e77c7b9928be/vllm/platforms/ init .py L231-L237 : TPU, CUDA, ROCm, XPU, and CPU. vLLM also loads any installed package registered under vllm.platform plugins https://github.com/vllm-project/vllm/blob/5df63a7203cdb9328d824a27b682e77c7b9928be/vllm/platforms/ init .py L250-L286 . vllm-ascend joins as ascend = vllm ascend:register https://github.com/vllm-project/vllm-ascend/blob/8fde6d4e335fe300e2b949aab0e31659cce0811d/setup.py L507 , which returns NPUPlatform https://github.com/vllm-project/vllm-ascend/blob/8fde6d4e335fe300e2b949aab0e31659cce0811d/vllm ascend/ init .py L72-L75 . The platform sets device name to "npu" and uses PrivateUse1 https://github.com/vllm-project/vllm-ascend/blob/8fde6d4e335fe300e2b949aab0e31659cce0811d/vllm ascend/platform.py L89-L99 , the PyTorch dispatch key reserved for devices that ship out of tree. It carries its own models/deepseek v41/ https://github.com/vllm-project/vllm-ascend/blob/8fde6d4e335fe300e2b949aab0e31659cce0811d/vllm ascend/models/deepseek v41/model.py and a patch/ package that edits vLLM classes at startup. The plugin is versioned alongside vLLM, and the tags we found were v0.30.0rc2 against v0.32.0 . Choosing Ascend on this side means following both repositories. vLLM already registers DeepseekV4ForCausalLM https://github.com/vllm-project/vllm/blob/5df63a7203cdb9328d824a27b682e77c7b9928be/vllm/model executor/models/registry.py L96 and DeepseekV41ForCausalLM https://github.com/vllm-project/vllm/blob/5df63a7203cdb9328d824a27b682e77c7b9928be/vllm/model executor/models/registry.py L373-L376 , with per-vendor code under vllm/models/deepseek v41/ for NVIDIA and AMD. DeepInfer’s list is the one model. “Engram” shows up in DeepInfer’s parallel state https://github.com/deepseek-ai/DeepInfer/blob/774ce244b87cc0eb8d4b94ab9f5de2ab22049f01/deepinfer/model executor/parallel utils/parallel state.py L70-L118 with no explanation beside it. The clearest note we found is vLLM’s, “n-gram hash lookups gated into the hyper-connection stream” https://github.com/vllm-project/vllm/blob/5df63a7203cdb9328d824a27b682e77c7b9928be/vllm/models/deepseek v41/common/engram.py L3-L8 , ported from DeepSeek’s reference code. ParallelConfig is the wider menu: tensor, pipeline, prefill-context, and data parallel sizes https://github.com/vllm-project/vllm/blob/5df63a7203cdb9328d824a27b682e77c7b9928be/vllm/config/parallel.py L139-L148 , enable expert parallel https://github.com/vllm-project/vllm/blob/5df63a7203cdb9328d824a27b682e77c7b9928be/vllm/config/parallel.py L189-L190 , enable eplb https://github.com/vllm-project/vllm/blob/5df63a7203cdb9328d824a27b682e77c7b9928be/vllm/config/parallel.py L205-L207 , and decode-context parallelism https://github.com/vllm-project/vllm/blob/5df63a7203cdb9328d824a27b682e77c7b9928be/vllm/config/parallel.py L385-L387 . We searched DeepInfer for tensor or pipeline parallel names and found none. DeepSeek’s deployment Q&A https://github.com/deepseek-ai/DeepInfer/blob/774ce244b87cc0eb8d4b94ab9f5de2ab22049f01/doc/inference deployment qa.md L53 says production does not use tensor parallelism. The groups here are expert, data, and context, plus engram. A tensor or pipeline setting from a vLLM config has nothing to map onto. Expert load balancing, copying busy experts onto more GPUs, is where the credit runs the other way. vLLM’s default policy says it was adapted from DeepSeek’s EPLB https://github.com/vllm-project/vllm/blob/5df63a7203cdb9328d824a27b682e77c7b9928be/vllm/distributed/eplb/policy/default.py L3-L11 . DeepInfer has its own greedy two-stage replication and placement https://github.com/deepseek-ai/DeepInfer/blob/774ce244b87cc0eb8d4b94ab9f5de2ab22049f01/deepinfer/model executor/eplb/policy.py L12-L120 . KV formats overlap and are spelled differently. vLLM’s cache dtype accepts fp8 e4m3 , fp8 e5m2 , fp8 ds mla , and nvfp4 ds mla https://github.com/vllm-project/vllm/blob/5df63a7203cdb9328d824a27b682e77c7b9928be/vllm/config/cache.py L39-L49 , and quantization includes deepseek v4 fp8 https://github.com/vllm-project/vllm/blob/5df63a7203cdb9328d824a27b682e77c7b9928be/vllm/model executor/layers/quantization/ init .py L38 . DeepInfer declares five cache formats https://github.com/deepseek-ai/DeepInfer/blob/774ce244b87cc0eb8d4b94ab9f5de2ab22049f01/deepinfer/model executor/layers/tilelang ops/cache common.py L109-L155 for its one model, with block scales stored per 16, 32 or 128 elements. Speculative verification Speculative decoding drafts several tokens with a cheap model, then checks them in one forward pass of the full model. DeepInfer sizes that check in C++ beside the scheduler. vLLM sizes it in Python on the GPU worker. Both take that size from the draft model’s own confidence. A ConfidenceHead https://github.com/deepseek-ai/DeepInfer/blob/774ce244b87cc0eb8d4b94ab9f5de2ab22049f01/deepinfer/model executor/models/mtp/dflash.py L46-L79 predicts acceptance per draft token. MTPVarlen sorts pending drafts by that rate https://github.com/deepseek-ai/DeepInfer/blob/774ce244b87cc0eb8d4b94ab9f5de2ab22049f01/csrc/engine/mtp varlen.hpp L213-L214 and keeps the batch size with the most expected tokens per predicted step time https://github.com/deepseek-ai/DeepInfer/blob/774ce244b87cc0eb8d4b94ab9f5de2ab22049f01/csrc/engine/mtp varlen.hpp L242-L249 . Step time is a median over a 5,000-step window https://github.com/deepseek-ai/DeepInfer/blob/774ce244b87cc0eb8d4b94ab9f5de2ab22049f01/csrc/engine/mtp varlen.hpp L48-L131 , so the cost is whatever this process has actually been paying. The path is opt-in, behind DEEPINFER ENABLE MTP VARLEN https://github.com/deepseek-ai/DeepInfer/blob/774ce244b87cc0eb8d4b94ab9f5de2ab22049f01/deepinfer/environ.py L129-L131 . AdaptiveVerificationManager scores each draft slot by the running product of confidences https://github.com/vllm-project/vllm/blob/5df63a7203cdb9328d824a27b682e77c7b9928be/vllm/v1/worker/gpu/spec decode/adaptive verification.py L309-L313 , prices the step from cost tables seeded by profiling runs at startup https://github.com/vllm-project/vllm/blob/5df63a7203cdb9328d824a27b682e77c7b9928be/vllm/v1/worker/gpu/spec decode/adaptive verification.py L188-L230 , and takes the argmax of expected accepted tokens over that cost https://github.com/vllm-project/vllm/blob/5df63a7203cdb9328d824a27b682e77c7b9928be/vllm/v1/worker/gpu/spec decode/adaptive verification.py L325-L344 . It is off by default, and only for method="dspark" https://github.com/vllm-project/vllm/blob/5df63a7203cdb9328d824a27b682e77c7b9928be/vllm/config/speculative.py L551-L553 . DeepSeek’s Q&A, which we did not reproduce, says this cuts draft tokens verified per step by about 30%, from 5.00 to 3.2–4.4, with accepted length roughly unchanged https://github.com/deepseek-ai/DeepInfer/blob/774ce244b87cc0eb8d4b94ab9f5de2ab22049f01/doc/inference deployment qa.md L247 , and estimates +4–7% throughput at about 60 concurrent sequences per rank from EP64/DP64 benchmark data. The table there https://github.com/deepseek-ai/DeepInfer/blob/774ce244b87cc0eb8d4b94ab9f5de2ab22049f01/doc/inference deployment qa.md L249-L256 shows a 1.5% loss at 16 sequences per rank. Those figures are DeepSeek’s. We found no comparable figure in vLLM’s source. Both use torch’s sync debug mode, which is process-wide. DeepInfer wraps forward and sample in forbid cuda sync https://github.com/deepseek-ai/DeepInfer/blob/774ce244b87cc0eb8d4b94ab9f5de2ab22049f01/deepinfer/worker/model runner.py L322-L325 and raises unless DEEPINFER DISABLE SYNC GUARD is set https://github.com/deepseek-ai/DeepInfer/blob/774ce244b87cc0eb8d4b94ab9f5de2ab22049f01/deepinfer/utils.py L119-L132 . vLLM’s gpu sync debug https://github.com/vllm-project/vllm/blob/5df63a7203cdb9328d824a27b682e77c7b9928be/vllm/utils/gpu sync debug.py L3-L16 catches those syncs and non-blocking copies through pageable memory, arms only after warmup https://github.com/vllm-project/vllm/blob/5df63a7203cdb9328d824a27b682e77c7b9928be/vllm/utils/gpu sync debug.py L51-L63 , and stays off unless VLLM GPU SYNC CHECK is warn or error https://github.com/vllm-project/vllm/blob/5df63a7203cdb9328d824a27b682e77c7b9928be/vllm/envs.py L656-L658 . What the credit line still matches The headers never name a vLLM version. We compared each of the 23 files with its counterpart at 16 tags from v0.2.0 to v0.32.0 and counted how many of DeepInfer’s non-comment lines still appear verbatim. It is a rough measure. Sixteen of the 23 are closest to a release from v0.2.0 to v0.5.0 , and the median overlap is 19%. The nearest are the shared-memory broadcast queue, 73% and best against v0.6.6 and v0.7.3 , and model executor/models/ init .py at 70%. Most of the rest kept the filename and rewrote the body. The buffer-moving bits still look like vLLM. The engine around them is DeepInfer’s. vLLM v0.3.3 ’s CacheEngine https://github.com/vllm-project/vllm/blob/v0.3.3/vllm/worker/cache engine.py L141-L144 is 178 lines and swaps blocks to CPU. DeepInfer’s is 978 lines, does not swap, and adds a KVCacheManager https://github.com/deepseek-ai/DeepInfer/blob/774ce244b87cc0eb8d4b94ab9f5de2ab22049f01/deepinfer/worker/cache engine.py L309 plus sliding-window state https://github.com/deepseek-ai/DeepInfer/blob/774ce244b87cc0eb8d4b94ab9f5de2ab22049f01/deepinfer/worker/cache engine.py L683 . sequence.py replaces vLLM’s sequence-group classes with MTPPredictInfo https://github.com/deepseek-ai/DeepInfer/blob/774ce244b87cc0eb8d4b94ab9f5de2ab22049f01/deepinfer/sequence.py L77 and SampledInfo https://github.com/deepseek-ai/DeepInfer/blob/774ce244b87cc0eb8d4b94ab9f5de2ab22049f01/deepinfer/sequence.py L114 . sampling params.py removes beam search https://github.com/vllm-project/vllm/blob/v0.2.7/vllm/sampling params.py L104 . Two of the 23 have no counterpart at v0.32.0 , because vLLM’s V0 worker/ directory is gone. The percentages are how you tell a small patch from a file that grew up. Use the GitHits API Each link above is a file and a line at the revision we read, the same jump the API gives an agent. The API overview https://docs.githits.com/api/overview maps those calls. Create a key in the token settings of your GitHits account https://app.githits.com/ and send it as a bearer token. The authentication guide https://docs.githits.com/api/authentication covers storing that key for CI, and the quickstart https://docs.githits.com/api/quickstart is the first request. If you’re a software factory interested in using our API, we’d love for you to book a call with us https://calendly.com/d/dvxq-7dn-msq/meeting-with-githits-select-15-or-30-minutes to discuss what it would look like.