A technical breakdown of how an LLM request moves through routing, prefill, decode, and KV caching, and why inference costs dwarf training over time.
What happens when you send a prompt to an LLM? #
A request to a large language model goes through a pipeline, not a single function call. The system tokenizes your prompt, checks whether any of it is already cached, runs a compute-heavy step called prefill to process the prompt, then enters a decode loop that produces output tokens one at a time. Each stage has different bottlenecks, and the gap between theoretical GPU performance and what you actually get at each stage is the central engineering problem in serving models at scale.
TL;DR #
- Inference, not training, dominates lifetime compute cost for a deployed model: DeepSeek’s own published numbers suggest inference flops can run roughly 6 to 12 times higher than training flops over a model’s useful life.
- Prefill and decode are fundamentally different workloads. Prefill processes the entire prompt in parallel and can hit high throughput, while decode generates one token at a time and is far less efficient per GPU.
- The KV cache is the single biggest lever for cost and latency. Reusing cached key-value states instead of recomputing them avoids rerunning prefill on repeated or shared context.
- Hardware is rarely used anywhere near its rated capacity. Model flop utilization during prefill might land around 50% of peak, and during decode it can drop to roughly 5%.
- Three categories of optimization stack together : changes to the model architecture (mixture-of-experts, state space models, latent attention), kernel and runtime tricks (flash attention, paged attention, radix caching), and harness-level routing and caching strategies.
- Cache hit rate depends heavily on traffic pattern. Agentic or coding workloads with repeated context get very high cache reuse, while one-off chat queries get almost none.
#
Plans first. Then code.
Remy writes the spec, manages the build, and ships the app.
Why does inference matter more than training? #
Training a model is a one-time (or periodic) cost. Serving it is ongoing, and every user query adds to that bill. DeepSeek is one of the few labs that has published enough detail to compare the two directly: training DeepSeek V3 took on the order of 2.8 million H800 GPU hours, while the company reported running roughly 43,000 H800 hours per day in inference. Annualized, that implies inference consumes something like 6 times the compute of training for that single weight file, and if the model stays in production for a couple of years, the ratio looks closer to 12x. As more users adopt a given model, that ratio only grows.
This has a direct implication for how models get evaluated and compared. Research papers often optimize and report results holding training flops constant (the Chinchilla-style approach). But if inference is the larger long-term cost, the more relevant comparison is holding inference flops constant instead, something the field has mostly not standardized around yet.
It also explains margins. Public pricing data suggests DeepSeek’s API margins can run as high as 80%, achieved specifically through inference-side optimization rather than any change to the underlying training recipe.
How does prefill work? #
Prefill is the step where the model processes your entire input prompt before generating anything. Because the whole prompt is known upfront, this step parallelizes well: the model can compute attention and activations across all input tokens simultaneously, which is why prefill throughput per GPU can be dramatically higher than decode throughput. Industry figures cited for a production serving stack put prefill throughput (excluding cache hits) around 60,000 tokens per second per GPU, and over 1.5 million tokens per second per GPU when cache hits are included.
Prefill is also where “time to first token” (TTFT) is determined: how long a user waits before the model starts streaming a response. Reported figures put median TTFT around 400 milliseconds, with the 99th percentile closer to 1 second, numbers that shift heavily based on prompt length and cache hit rate.
If part of a prompt has already been processed before (a repeated system prompt, a shared conversation prefix, a tool definition reused across calls), a well-built serving system shouldn’t recompute it. That’s the point of the KV cache.
What is the KV cache and why does it matter so much? #
During attention, a model computes “key” and “value” vectors for every token it has seen. The KV cache stores these so they don’t need to be recalculated on every subsequent step. Without it, generating each new token would require reprocessing the entire conversation history from scratch.
- ✕a coding agent
- ✕no-code
- ✕vibe coding
- ✕a faster Cursor
The one that tells the coding agents what to build.
Caching turns into a systems problem once you’re serving millions of requests. Providers keep recently used KV states hot in GPU memory (DRAM) and may fall back to something like a Redis layer or SSD storage for less recently used entries, evicting old entries with an LRU-style (least recently used) policy. One serving provider reportedly retains KV cache data for up to 72 hours across petabytes of stored state, which is how they push input cache hit rates as high as 96%.
Cache effectiveness depends entirely on traffic shape. Agentic workflows and coding assistants that repeatedly reuse large chunks of context (a codebase, a system prompt, tool schemas) get strong cache reuse and cheaper, faster prefill. A chat product where every user starts a brand-new, unrelated conversation gets almost no reuse, and every request pays the full prefill cost.
How does decode differ from prefill, and why is it the bottleneck? #
Decode is the autoregressive loop: generate one token, feed it back in, generate the next one, repeat. Unlike prefill, this step can’t be parallelized across the sequence, because each new token depends on the one before it. That sequential dependency is why decode throughput lags prefill throughput so sharply: where prefill might hit 1.5 million tokens per second per GPU with cache hits, decode throughput on the same hardware might sit around 14,000 tokens per second per GPU.
This gap shows up in GPU utilization numbers too. Model flop utilization (MFU), the share of a chip’s advertised peak compute that you actually realize in practice, might land around 50% during prefill and training, but can fall to roughly 5% during decode. A GPU rated for a given number of FP8 flops is nowhere near that figure once you’re generating tokens one at a time. This is also why so much chip and systems innovation over the past few years has specifically targeted decode efficiency rather than prefill.
The relevant metrics teams track here are time per output token (how long each decode step takes), throughput (tokens produced per GPU-hour), and “goodput” (tokens delivered within an acceptable latency target, not just raw throughput).
What techniques actually make inference faster? #
Optimizations tend to fall into three buckets.
Model-level changes alter the architecture itself. Mixture-of-experts (MoE) models route each token through only a subset of available experts, cutting the effective compute per token. State space models and hybrid architectures reduce how often full attention needs to run across layers. Multi-head latent attention compresses representations along the sequence dimension to reduce flops. Quantization (running models in lower-precision formats, sometimes as low as 2 bytes per parameter) and distillation (training a smaller model to mimic a larger one) also fall here.
Kernel and runtime innovations optimize how computation actually executes on hardware, often borrowing directly from classic operating systems and database theory. Flash attention tiles computation to better exploit GPU memory hierarchy. Paged attention applies virtual-memory-style paging to how KV cache data moves in and out of GPU memory. Radix attention uses prefix-tree-style structures to identify and reuse shared context across requests.
Everyone else built a construction worker.
We built the contractor.
One file at a time.
UI, API, database, deploy.
Harness-level strategies sit above the model entirely. Routing sends simpler queries to smaller, cheaper models and reserves larger models for harder requests. Speculative decoding uses a smaller “draft” model to propose multiple tokens that a larger model then verifies in parallel, trading some wasted compute for lower latency. Context-management approaches, sometimes framed as recursive or selective context , aim to avoid stuffing the entire conversation history into the KV cache by letting the model decide what context it actually needs.
Is inference optimization worth the engineering investment? #
For any team serving models at meaningful scale, yes. Inference costs compound with every user and every request, in a way training costs don’t. A serving stack with poor cache reuse, low MFU, and naive routing will cost multiples more per token than one that’s been tuned, and that gap shows up directly in gross margin for anyone running an API business on top of these models. For teams building applications rather than serving infrastructure, understanding this pipeline still matters: prompt structure, context reuse, and traffic patterns directly affect latency and cost, even when you’re not the one running the GPUs.
Frequently Asked Questions #
What is the difference between prefill and decode?
Prefill processes the entire input prompt at once and can be heavily parallelized, giving it much higher throughput. Decode generates output tokens one at a time, where each token depends on the previous one, making it sequential and far less GPU-efficient.
What does “time to first token” mean?
Time to first token (TTFT) measures how long a user waits between sending a request and receiving the first generated token. It’s determined largely by prefill speed and whether relevant context was already cached.
Why is the KV cache important for cost?
The KV cache stores previously computed attention states so they don’t need to be recalculated. High cache hit rates mean a provider can skip expensive prefill computation for repeated or shared context, which directly lowers compute cost and latency.
Why is GPU utilization so low during decode?
Decode generates tokens sequentially, so each step can’t exploit the same parallelism prefill can. This leaves much of the GPU’s rated compute capacity idle, with reported model flop utilization during decode far lower than during prefill or training.
Does inference really cost more than training over time?
For a model deployed at scale, yes, based on figures DeepSeek has published. Training is a bounded, one-time cost, while inference scales with every query the model serves over its deployment lifetime, which can run many times higher than the original training compute.