- NewNew Published in the last 14 days
What an AI inference engine is and how it works, what DeepSeek released with its open-source DeepInfer engine, how it compares with vLLM, SGLang, TensorRT-LLM, llama.cpp and managed APIs, and what it means for leaders, developers and enterprises.
- 1
- model DeepInfer supports at launch: DeepSeek-V4.1-Flash
- 2
- hardware families: NVIDIA SM100 GPUs and Huawei Ascend 950 NPUs
- 3
- parts of a full service DeepSeek left out: KV cache system, disaggregation, load balancing
- Apache
- licence: Apache 2.0, free for commercial use
Most conversations about AI focus on models: which one is smartest, which tops a benchmark, which is cheapest per token. Far less attention goes to the software that actually runs those models in production. Yet that layer, the inference engine, often decides whether an AI feature responds in half a second or five, and whether it costs a few hundred dollars a month or tens of thousands.
This guide explains what an inference engine is in plain English, how it works, what DeepSeek released with DeepInfer, how it compares with the engines most teams use today, and what all of it means for different kinds of organisations, from start-ups and agencies to enterprises with strict data rules.
What is an AI inference engine? #
Building an AI model has two phases. Training is when the model learns from data, a huge one-off job. Inference is when the trained model is used: you send it a prompt and it produces an answer. Every chatbot reply, code suggestion and document summary is an inference request.
A model on its own is just a very large file of numbers, its weights. To answer a question, something has to load those weights onto hardware, feed in the prompt, run billions of calculations, pick the next word, and repeat until the answer is complete, all while serving many users at once. That something is the inference engine.
A useful comparison is a database. The data is valuable, but it is the database engine, such as PostgreSQL or MySQL, that decides how queries run, how memory is used and how many users it can serve. Two companies with the same data can see very different performance depending on the engine and how it is tuned. Models and inference engines work the same way.
Two phases of AI
Training vs inference at a glance #
Training
The model learns
- What happens
- The model learns patterns from huge amounts of data
- How often
- Once per model version
- Who pays
- Mostly the AI lab
- What decides cost
- GPU hours for weeks or months
- Software that runs it
- Training frameworks such as PyTorch
Inference
The model answers
- What happens
- The trained model answers each prompt
- How often
- Every time anyone uses the product
- Who pays
- You, per token or per GPU
- What decides cost
- Usage, latency targets and engine efficiency
- Software that runs it
- An inference engine
How an inference engine works #
Large language models generate text one token (roughly a word fragment) at a time. Each request goes through two stages, and most engine design is about making both efficient.
- Prefill. The engine reads the whole prompt at once and builds an internal memory of it, called the KV cache. This is heavy on computation and decides how long the user waits for the first word.
- Decode. The engine produces the answer token by token, each step reading the KV cache. This is limited by memory speed and decides how fast the answer streams out.
Inside the engine
From prompt to answer in five steps #
-
Queue and batchYour request joins others; the scheduler groups them so the GPU stays busy.
-
TokeniseThe prompt is split into tokens, the word fragments the model works with.
-
PrefillThe whole prompt is processed at once and stored in the KV cache. This sets the wait for the first word.
-
DecodeThe answer is produced one token at a time, reading the KV cache each step. This sets streaming speed.
-
Stream and freeTokens stream back to the user; when the answer ends, memory is released for the next request.
On top of that, a production engine juggles hundreds or thousands of requests of different lengths at the same time. The main techniques it uses are:
- Continuous batching
- KV cache reuse
- Quantisation
- Speculative decoding
- Parallelism across chips
- Optimised kernels
| Technique | What it does | Why it matters |
|---|---|---|
| Continuous batching | Adds and removes requests from the running batch at every step instead of waiting for a batch to finish | Keeps expensive GPUs busy, often the biggest single throughput gain |
| KV cache management | Stores and reuses the memory of each conversation efficiently, sometimes shared between requests with the same prefix | Lets more users fit on the same hardware and speeds up repeated prompts |
| Quantisation | Runs the model with smaller numbers, such as 8-bit or 4-bit, instead of 16-bit | Cuts memory and cost, sometimes at a small quality cost |
| Speculative decoding | A quick draft predicts several tokens ahead and the full model checks them in one step | Faster answers with the same output quality |
| Parallelism | Splits a model too big for one chip across many, by layer, tensor or expert | Makes very large models servable at all |
| Optimised kernels | Hand-tuned low-level code for attention and matrix maths on specific chips | Squeezes out performance that general code leaves on the table |
| Prefill/decode disaggregation | Runs the two stages on separate machine pools sized for each | Better latency and cost at large scale |
None of these change what the model knows. They change how fast, how cheaply and how reliably it answers, which is exactly what users and finance teams notice.
Why inference engines matter now #
Three shifts have pushed inference from a technical detail to a board-level cost line.
- Inference is where the money goes. Training happens once per model version; inference runs every time anyone uses the product. For a successful AI feature, inference spending keeps growing with usage.
- Open-weight models are good enough for many jobs. Models from DeepSeek, Meta, Mistral, Alibaba and others can be downloaded and run on your own infrastructure. Once you run a model yourself, the engine choice is yours too.
- Agents multiply the number of calls. AnAI agent may call a model dozens of times to complete one task. Small gains in latency and cost per call compound quickly, as we discuss in our guide toscaling agentic AI without agent sprawl .
DeepSeek DeepInfer: what was released #
In October 2026 DeepSeek published DeepInfer on GitHub under the Apache 2.0 licence. DeepSeek describes it as the high-performance inference framework that powers its own large-scale production services. Within about a month the repository had passed 26,000 stars, a sign of how much interest there is in seeing how a frontier AI lab actually serves its models.
What it includes
- Optimised kernels from DeepSeek's earlier open-source projects, including FlashMLA, DeepGEMM and DeepEP, plus new TileLang kernels.
- Parallelism for mixture-of-experts models: attention data parallelism, balanced context parallelism, and expert parallelism with load balancing (EPLB).
- Faster decoding through DSpark draft tokens with confidence-scheduled verification, a form of speculative decoding.
- Asynchronous scheduling that overlaps CPU batch planning with GPU work, enabled by default.
- Batch-invariant inference and deterministic sampling on NVIDIA, so the same input gives the same output regardless of what else is in the batch. This is valuable for testing, audits and regulated use.
- Structured outputs constrained to a regular expression, JSON schema or grammar, which makes integration with other software far more reliable.
What it does not include
DeepSeek is unusually clear about the gaps. The README lists three things required for a complete inference service that are not in this release:
- A KV cache system.
- Disaggregation of the vision encoder, prefill and decode stages, although most of the engine functionality for it is in place.
- Load balancing across data-parallel replicas for both prefill and decode.
The release also supports a single model, DeepSeek-V4.1-Flash, and two hardware families: NVIDIA SM100 GPUs (the Blackwell generation) and Huawei Ascend 950 NPUs. It needs Linux, Python 3.12 and CUDA 13 or CANN 9.2, and the supported entry point is offline inference: a script that processes prompts rather than a ready-made API server. DeepSeek's own benchmark charts were measured on 4 GPUs for prefill and 64 for decode.
In other words, DeepInfer is the engine of a race car, not the car. It shows how DeepSeek gets its performance, but you would need to build the rest of the service around it, and you would need the latest data-centre hardware to run it at all.
Included in DeepInfer
- Optimised kernels: FlashMLA, DeepGEMM, DeepEP, TileLang
- Attention data parallelism and expert parallelism with load balancing
- DSpark speculative decoding with confidence-scheduled verification
- Asynchronous, zero-overhead batch scheduling
- Batch-invariant inference and deterministic sampling (NVIDIA)
- Structured outputs: regex, JSON schema, grammar
Not included (yet)
- A KV cache system
- Vision encoder, prefill and decode disaggregation
- Load balancing across data-parallel replicas
- An OpenAI-compatible API server
- Models other than DeepSeek-V4.1-Flash
- Hardware older than NVIDIA SM100 or other than Ascend 950
Why the release still matters
- Transparency. It is rare for a major AI provider to publish the engine behind its API. Researchers and engine teams can now study real production techniques, not just papers.
- Influence on the tools you will use. DeepInfer was built primarily on vLLM, with ideas from SGLang and TensorRT-LLM . Expect its best ideas to flow back into those projects, which is how most teams will benefit.
- Hardware choice. First-class support for Huawei Ascend alongside NVIDIA shows serious inference software is no longer NVIDIA-only, which matters for supply, price and regional regulation.
- Determinism as a feature. Reproducible outputs are moving from research curiosity to a production requirement, especially for testing and compliance.
How DeepInfer compares with other inference engines #
The table compares DeepInfer with the engines and tools teams most often consider. It reflects each project's documented scope at the time of writing rather than benchmark numbers, which depend heavily on model, hardware and workload.
Self-hosted engines
DeepInfer vs the popular alternatives #
| DeepInferDeepSeek, Oct 2026 | vLLMThe general default | SGLangThroughput and agents | llama.cpp / OllamaLocal and edge | |
|---|---|---|---|---|
| Best for | DeepSeek-V4.1-Flash at very large scale | Self-hosting most open models | Agents and shared-prefix workloads | Laptops, desktops and private local use |
| Models | One | Hundreds | Most popular open models | Many, quantised GGUF |
| Hardware | NVIDIA SM100, Huawei Ascend 950 | NVIDIA, AMD, Intel, TPU and more | NVIDIA, AMD and more | CPU, Apple silicon, consumer GPUs |
| API server included | No | Yes OpenAI-compatible | Yes OpenAI-compatible | Yes small scale |
| Ease of getting started | Hard: build from source, latest GPUs | Easy: pip install | Moderate | Very easy |
DeepInfer
DeepSeek, Oct 2026
-
Best for
-
DeepSeek-V4.1-Flash at very large scale
-
Models
-
One
-
Hardware
-
NVIDIA SM100, Huawei Ascend 950
-
API server included
-
No
-
Ease of getting started
-
Hard: build from source, latest GPUs
vLLM
The general default
-
Best for
-
Self-hosting most open models
-
Models
-
Hundreds
-
Hardware
-
NVIDIA, AMD, Intel, TPU and more
-
API server included
-
Yes OpenAI-compatible
-
Ease of getting started
-
Easy: pip install
SGLang
Throughput and agents
- Best for
- Agents and shared-prefix workloads
- Models
- Most popular open models
- Hardware
- NVIDIA, AMD and more
- API server included
- Yes OpenAI-compatible
- Ease of getting started
- Moderate
llama.cpp / Ollama
Local and edge
- Best for
- Laptops, desktops and private local use
- Models
- Many, quantised GGUF
- Hardware
- CPU, Apple silicon, consumer GPUs
- API server included
- Yes small scale
- Ease of getting started
- Very easy
| Engine | Best for | Models | Hardware | Ready to serve an API? |
|---|---|---|---|---|
| DeepInfer (DeepSeek) | Studying or running DeepSeek-V4.1-Flash at very large scale | One: DeepSeek-V4.1-Flash | NVIDIA SM100, Huawei Ascend 950 | No: offline inference; KV cache, disaggregation and load balancing not included |
| vLLM | General-purpose self-hosting of open models | Hundreds of open models | NVIDIA, AMD, Intel, TPU and others | Yes: OpenAI-compatible server |
| SGLang | High-throughput serving, agents and prompts with shared prefixes | Most popular open models | NVIDIA, AMD and others | Yes: OpenAI-compatible server |
| NVIDIA TensorRT-LLM | Maximum performance on NVIDIA hardware | Major open models, compiled per GPU | NVIDIA only | Yes, usually with NVIDIA Triton or Dynamo |
| llama.cpp and Ollama | Laptops, desktops, edge devices and private local use | Many open models in quantised GGUF format | CPU, Apple silicon, consumer GPUs | Yes, for small-scale use |
| Managed APIs | Fastest start with no infrastructure | Provider's own and hosted open models | Provider's | Yes: pay per token |
The pattern is clear. DeepInfer is the most specialised option on the list: one model, the newest hardware and no serving layer. vLLM and SGLang are the practical defaults for self-hosting. TensorRT-LLM suits teams fully committed to NVIDIA that want every last bit of performance. llama.cpp and Ollama cover local and edge use. Managed APIs remain the right start for most new projects.
How inference engines are useful for different users #
At a glance
Which route fits you? #
-
Start-up or product teamNew AI feature, uncertain usage, small team.
- Measure
- Cost per request and time to first token.
- Decide
- Managed API now; vLLM or SGLang when usage is steady and high.
-
Enterprise with data rulesPrompts contain personal or confidential data that must stay in-house.
- Measure
- Data residency, audit needs, GPU utilisation.
- Decide
- Self-host with vLLM or SGLang on your own cloud or data centre.
-
Agency or web teamAI features inside WordPress, Drupal, Laravel or PHP products.
- Measure
- Structured output reliability, streaming, cost per client.
- Decide
- API behind a standard interface; choose providers with JSON outputs and prompt caching.
-
AI platform team at scaleMillions of requests, latest GPUs, mixture-of-experts models.
- Measure
- Throughput per GPU, tail latency, reproducibility.
- Decide
- Evaluate DeepInfer techniques; benchmark against SGLang and TensorRT-LLM.
Business leaders and CTOs
You do not need to choose kernels, but you do need to know that the inference layer is a cost and risk decision. The questions that matter are: are we paying per token or for capacity, at what usage does self-hosting become cheaper, and who would run it? Our guide to evaluating the ROI of AI automation shows how to put numbers on that before committing.
Start-ups and product teams
Start with a managed API. It gets a feature in front of customers in days, and usage data will tell you whether self-hosting is worth it. Keep your code behind an OpenAI-compatible interface so you can switch to vLLM or SGLang later without rewriting the product.
Enterprises with data, privacy or residency rules
Self-hosted inference keeps prompts and data inside your own cloud account or data centre, which can settle legal and compliance questions that block AI projects. vLLM or SGLang on managed GPUs is the usual route. DeepInfer's deterministic mode points to where audit-friendly AI is going, even if you do not run DeepInfer itself.
Agencies and web development teams
For agencies building on PHP, WordPress, Drupal or Laravel, the engine is usually hidden behind an API. What matters is features that engines now provide: structured JSON outputs for reliable integrations, streaming for responsive interfaces and caching of shared prompt prefixes to cut cost. Choose providers and self-hosted stacks that support these.
AI engineers and platform teams
This is the group DeepInfer serves most directly. Study its scheduling, speculative decoding and mixture-of-experts parallelism, benchmark your own workloads on vLLM and SGLang, and watch for DeepInfer techniques arriving upstream. Teams moving pilots to production should read our guide on moving AI pilots into production, where inference cost and latency often decide success.
Researchers and students
DeepInfer is a rare look at a production-grade engine from a frontier lab, with a licence that allows study and reuse. For learning on modest hardware, llama.cpp and Ollama remain the most accessible way to see inference concepts in action.
Individuals and hobbyists
Ollama or llama.cpp can run capable open models on a recent laptop, with no data leaving the machine. DeepInfer is not for this audience: it needs data-centre GPUs.
How to choose an inference engine: a decision guide #
-
- Do we need to self-host? No clear cost, privacy or control reason → use a managed API.
-
- Which models? DeepSeek-V4.1-Flash at huge scale on SM100 or Ascend → evaluate DeepInfer. Anything else → look elsewhere.
-
- Which hardware? NVIDIA-only → TensorRT-LLM is an option. Mixed or non-NVIDIA → vLLM or SGLang. Laptops and edge → llama.cpp or Ollama.
-
- What workload? Agents and long shared prompts → prefix caching (SGLang). Long one-off documents → strong prefill.
-
- Reproducible outputs? Audits or testing need identical answers → check determinism support explicitly.
-
- Who runs it? Budget the people, upgrades, monitoring and on-call, not just the GPUs.
Common misconceptions
Inference engine myths vs reality #
- MythA better model always means faster answers. RealitySpeed and cost depend as much on the engine, hardware and batching as on the model.
- MythOpen-sourcing DeepInfer means anyone can run DeepSeek's API. RealityThe release leaves out the KV cache system, disaggregation and load balancing, and needs the newest data-centre chips.
- MythSelf-hosting is always cheaper than an API. RealityOnly with steady, high usage. Idle GPUs and the people to run them often cost more than tokens.
- MythVendor benchmark charts tell you which engine to pick. RealityResults swing with model, prompt length and concurrency. Test with your own traffic.
Common mistakes to avoid #
- Comparing engines on vendor charts alone. Benchmarks depend on model, hardware, prompt length and concurrency. Test with your own traffic.
- Self-hosting too early. GPU capacity sits idle when traffic is low. Managed APIs are usually cheaper until usage is steady and high.
- Ignoring latency to the first token. Users judge speed by how quickly an answer starts, not by total throughput.
- Locking code to one provider. Use a standard interface so you can change models and engines as prices and quality shift.
- Forgetting the rest of the service. As DeepSeek's own release shows, an engine is only one part. Caching, load balancing, monitoring and security make it a service.
What to watch next #
Expect DeepSeek to add more models and serving components to DeepInfer, techniques such as confidence-scheduled speculative decoding and batch-invariant inference to spread to vLLM and SGLang, and wider competition between NVIDIA and other chip makers for inference workloads. For most organisations, the practical result will be faster, cheaper and more predictable AI from the tools they already use.
Sources #
FAQ
Frequently asked questions #
5 questions readers ask about this topic
01What is an AI inference engine? #
An AI inference engine is the software that runs a trained model to produce answers from prompts. It handles batching, memory, scheduling and spreading work across GPUs, which decides how fast and how cheaply the model responds.
02What is DeepSeek DeepInfer? #
DeepInfer is the inference framework behind DeepSeek's production services, open-sourced in October 2026 under Apache 2.0. It supports DeepSeek-V4.1-Flash on NVIDIA SM100 GPUs and Huawei Ascend 950 NPUs, for offline inference.
03Is DeepInfer better than vLLM? #
For most teams, no. DeepInfer is specialised for one model on the newest hardware and lacks a full serving layer. vLLM supports hundreds of models on many kinds of hardware with an OpenAI-compatible server, so it remains the practical default for self-hosting.
04What is the difference between training and inference? #
Training teaches a model from data and happens once per model version. Inference uses the trained model to answer requests and runs every time someone uses the product, so it usually becomes the larger ongoing cost.
05Should my company self-host AI inference? #
Self-host when steady, high usage makes it cheaper than an API, or when data must stay in your own infrastructure. Otherwise start with a managed API and keep your code on a standard interface so you can move later.
Still have a question?Ask me directly