DeepSeek's own numbers show inference can outweigh training compute 12x over a model's life. Here's why that changes how you pick models.
The core question: does training or inference compute cost more over a model’s life? #
Inference wins, by a wide margin. Using DeepSeek’s own published numbers, training V3 took roughly 2.8 million H800 GPU hours, while the company runs about 43,000 H800 hours per day serving that same model. Annualized, that’s already around 6 times more compute spent on inference than on training. Stretch the model’s useful life to two years, which is realistic for a popular open weight file, and inference compute runs about 12 times higher than training compute. As more users adopt a given model, that ratio only grows.
TL;DR #
- DeepSeek is one of the only labs that publishes enough data to actually compare training flops against inference flops, and the comparison favors inference heavily.
- Inference compute outpaces training compute by roughly 6x per year for DeepSeek V3, and by around 12x over an assumed two year model lifespan.
- Benchmark papers mostly optimize the wrong axis , reporting results at fixed training flops (in the style of the Chinchilla paper) instead of fixed inference flops.
- Serving efficiency directly determines margins , and DeepSeek’s own numbers imply gross margins near 80% largely because of inference side optimization.
- Model flop utilization is brutal in practice : prefill might hit around 50% of a GPU’s rated flops, while decode can fall to roughly 5%.
- Three levers drive inference efficiency : model level changes (MoE, SSMs, quantization), kernel and runtime tricks (flash attention, paged attention, radix caching), and harness level decisions (routing, speculative decoding, context management).
- KV cache retention policy is a quiet lever on cost , since holding cached context for hours at a time (DeepSeek reportedly keeps it around for 72 hours) is what makes high cache hit rates, and high margins, possible.
Remy doesn't write the code. It manages the agents who do. #
Remy runs the project. The specialists do the work. You work with the PM, not the implementers.
Why does inference compute eventually dwarf training compute? #
A model gets trained once. It gets queried, potentially, billions of times. Training is a fixed cost. Inference is a cost that scales with usage, and usage only grows as a model becomes popular, gets embedded into products, or powers agents that call it repeatedly throughout a workflow.
DeepSeek’s disclosed figures put this in concrete terms: about 2.8 million H800 hours to train V3, against roughly 43,000 H800 hours burned every single day just to serve it. Run that daily number out over a year and you’re already well past the training number, by a factor of about 6x. Assume a two year useful life for the weight file, which is conservative for a model that stays in production, and the ratio climbs to around 12x.
That 12x isn’t a one-time fact about DeepSeek. It’s a structural feature of how deployed models work: the more successful a model is, the more lopsided the ratio gets, because inference volume keeps climbing while training compute is already spent and fixed.
What does this mean for how researchers evaluate models? #
Most benchmark papers, and most leaderboard-style comparisons, hold training flops constant when comparing models. This follows the convention set by papers like Chinchilla, which plot performance against training compute on the x-axis. That convention made sense when training was the dominant cost center. It makes much less sense now.
If inference compute matters 12 times more than training compute over a model’s life, then evaluations should increasingly hold inference flops constant (ISO inference flops) rather than training flops. A model that trains a bit less efficiently but serves far more cheaply per token is, by this framing, the better choice for almost any real deployment. Minimizing training flops is still worth doing. It’s just the secondary consideration.
How hard is it to actually serve inference efficiently? #
Harder than it looks. The core operation, the forward pass of a model, sounds simple. But production inference breaks into stages, each with its own bottlenecks.
A request first goes through prefill, where the model processes the initial prompt. If that prompt (or part of it) has already been processed before, a well built system should reuse the cached result rather than recomputing it, whether that cache lives in GPU memory, system RAM, or a layer like Redis. After prefill comes decode, the token by token generation loop that produces the actual output.
The efficiency gap between these stages is stark. Model flop utilization (MFU), the share of a chip’s advertised peak flops you actually get at inference time, might land around 50% during prefill. During decode it can drop to roughly 5%. A GPU rated for 2 trillion FP8 flops is, in practice, delivering a small fraction of that once you’re in the decode loop. This is why so much inference engineering effort concentrates on decode specifically.
Other agents ship a demo. Remy ships an app. #
Real backend. Real database. Real auth. Real plumbing. Remy has it all.
Teams track a specific set of metrics to manage this: time to first token (how long prefill takes), time per output token (the pace of the decode loop), throughput (tokens produced per GPU hour), and goodput (throughput that actually meets a service level target). Numbers shared publicly by providers illustrate the spread: prefill throughput excluding cache hits can run around 60,000 tokens per second per GPU, but with a high cache hit rate (figures near 96% have been reported) that number can jump into the range of 1.5 million tokens per second. Decode throughput, by contrast, tends to sit far lower, in the range of 14,000 tokens per second per GPU, which is the real bottleneck in serving at scale.
What actually reduces inference cost? #
Optimizations tend to fall into three buckets.
Model level changes. Mixture of Experts (MoE) architectures only activate a subset of experts per token, cutting the effective flops needed per forward pass. State space models (SSMs) can skip full attention on most layers, applying it only periodically instead of at every layer. Techniques like multi-headed latent attention compress information in the sequence dimension to shrink the compute and memory needed for attention. Quantization (running models in lower precision, down to very small bit widths) and distillation (training a smaller model to mimic a larger one) both reduce the cost of serving a given capability level.
Kernel and runtime innovations. Many of these borrow directly from classic operating systems and database theory. Flash attention tiles computation to use GPU memory more efficiently. Paged attention applies a virtual memory style paging scheme to how KV cache data gets managed in GPU memory. Radix attention uses prefix tree structures to identify and reuse shared context across requests, which is part of what drives high cache hit rates in production.
Harness level decisions. This is the layer closest to the application. Routing sends easy queries to smaller, cheaper models and reserves large models for queries that need them. Speculative decoding uses a smaller draft model to propose tokens that a larger model then verifies, speeding up generation. Careful context management, deciding what actually needs to live in the KV cache rather than stuffing the full conversation history in by default, can meaningfully cut memory and compute overhead. Compaction, trimming and summarizing context over time, is increasingly treated as a standard part of any serious inference harness.
Is inference efficiency really what drives model serving margins? #
Yes, and the numbers back it up. Published figures around DeepSeek’s serving operation imply gross margins near 80%, a number that surprised people when it came out. That margin isn’t primarily a function of hardware pricing. It’s a function of inference engineering: high cache hit rates, efficient decode throughput, and a KV cache retention policy (DeepSeek reportedly holds cached context for around 72 hours) that lets the system avoid recomputing prefill for repeat or related queries.
This is also why workload shape matters so much. A coding agent that repeatedly builds on the same context will get excellent cache hit rates and cheap serving economics. A consumer chat product where users constantly start fresh conversations gets little benefit from caching and looks far more expensive to serve per token, even on the same underlying model.
Frequently Asked Questions #
Why does inference compute exceed training compute for a popular model?
Training happens once, at a fixed compute cost. Inference happens every time a user or application queries the model, and that volume accumulates over the model’s entire deployment life, which for a widely used model can span years and billions of requests.
What is model flop utilization (MFU) and why does it matter?
- ✕a coding agent
- ✕no-code
- ✕vibe coding
- ✕a faster Cursor
The one that tells the coding agents what to build.
MFU is the percentage of a GPU’s advertised peak compute that a workload actually achieves. It matters because advertised hardware specs are theoretical maximums; real inference workloads, especially decode, often achieve only a small fraction of that ceiling, which directly affects serving cost.
Why is decode so much slower than prefill?
Prefill processes a batch of known tokens at once, which maps efficiently onto GPU compute. Decode generates one token at a time, which is memory bandwidth bound rather than compute bound, leading to much lower effective utilization and throughput.
How does KV cache retention affect cost?
Keeping recently used context cached in memory lets a system skip reprocessing (prefill) for repeated or related queries. Holding that cache for a longer window, such as the 72 hours reportedly used by DeepSeek, increases cache hit rates and lowers the average compute cost per request.
Should model comparisons focus on training flops or inference flops?
Given that inference compute dominates a model’s lifetime cost, comparisons that hold training flops constant miss the more important variable. Evaluating models at a fixed inference flop budget gives a more realistic picture of which model is actually cheaper and more practical to deploy at scale.