LLM inference is the process of running a trained large language model to turn a prompt into output tokens. The model reads your input, predicts the next token, and repeats until the response is complete. No learning happens during this step.
That is the one-line definition. But it hides everything interesting.
Every time you open ChatGPT, Claude, or Gemini and type something, a cascade of mathematics fires across thousands of GPU cores, a probability distribution forms over a vocabulary of 100,000 words, and within milliseconds a token appears on your screen. That process, repeated token by token, word by word, is inference. It is the moment AI stops being a research artifact and becomes a working product.
The inference market is projected to exceed $50 billion in 2026, growing faster than the training compute market for the first time. Companies spend more running models than building them. Inference now represents 55% of AI infrastructure spending in early 2026, up from 33% in 2023.
And yet, most developers who build on top of these models treat inference as a black box. They call an API, get a response, and move on. This guide is for people who want to open the box.
Let’s start at the very beginning.
Model inference is the phase where a trained AI model is deployed to make predictions or generate outputs based on new, unseen data. In simpler terms, inference is what happens after the model has learned from its training data. It is the act of putting the model to work.
The word “inference” itself is borrowed from logic. Setting AI aside for a second, inference is the logical process of drawing conclusions from existing evidence, observations, and background knowledge. In the context of AI, inference is the process by which a pre-trained model takes new, unseen data and produces a prediction or decision based on what it learned during training. Pre-trained here means the model has already completed training; its internal weights are adjusted to recognize patterns in the data, and those weights are fixed. The model is no longer learning. It is applying what it already knows.
The best analogy: training is the model studying for an exam. Inference is the model sitting the exam and answering questions it has not seen before, using what it learned during training.
Real-world examples of inference in action:
When you interact with an AI chatbot, ask it to summarize a document, translate text, or generate code, you are initiating the inference process. Customer support chatbots generate personalized, contextually relevant replies in real time. Writing assistants complete sentences, correct grammar, or summarize long documents. Developer tools convert natural language descriptions into executable code. AI agents perform complex, multi-step reasoning and decision-making processes autonomously.
In every one of those cases, the same mechanical process is running underneath.
Before going deeper into how inference works, you need to understand what makes it fundamentally different from training.
Training requires large datasets, significant GPU compute, and substantial processing time. The quality of training directly determines how well a model performs during inference. A model trained on low-quality or biased data will produce unreliable predictions no matter how well the inference infrastructure is optimized.
During training, the model sees enormous amounts of text. It runs a forward pass to generate predictions, computes a loss (how wrong it was), and then uses backpropagation to update billions of internal parameters. This loop runs for weeks across thousands of GPUs. Every parameter nudges slightly closer to producing correct outputs.
Once the model meets performance benchmarks, the trained parameters are saved, and the model is packaged for inference. At that point, the model is frozen; it will not learn anything new until it is retrained.
AI inference is the phase where a trained model is used to generate outputs from new, unseen inputs. Once training is complete, the model’s weights are frozen, and the system focuses entirely on efficient execution. There is no loss calculation, no backward pass, and no weight updates. This dramatically reduces computational complexity compared to training.
If training is “build the factory,” inference is “run the production line every day.” Most of the cost and most of the engineering complexity lives in the second part.
Here is the comparison in concrete terms:
The most counterintuitive thing about inference vs training is the cost ratio. Each individual training run is dramatically more expensive than each individual inference. But over the lifetime of a deployed model, inference wins by a wide margin.
OpenAI’s 2024 inference spend reached $2.3 billion: 15 times the training cost for GPT-4.
Inference will account for 80 to 90% of a model’s total compute costs in production. This is why every serious AI company in 2026 is obsessed with inference optimization. Shaving 10 milliseconds off latency or 20% off cost per token directly translates to hundreds of millions of dollars at scale.
Now let’s open the hood.
When you type “What is the capital of France?”, the model does not see words. It sees numbers. Actually, it generates tokens in response text one by one. Each token generation is a forward pass of the language model.
The first thing that happens is tokenization. Your text is broken into tokens, which are chunks of characters. A token is not always a word. “tokenization” might be two tokens: “token” and “ization”. Numbers, punctuation, and spaces all get tokenized too. Most modern LLMs use a vocabulary of roughly 100,000 to 200,000 tokens.
import tiktokenenc = tiktoken.get_encoding("cl100k_base") text = "What is the capital of France?"tokens = enc.encode(text)print(tokens)# Output: [3923, 374, 279, 6864, 315, 9822, 30]print(f"Token count: {len(tokens)}")# Output: Token count: 7
Each number maps to a specific token in the model’s vocabulary. These integer IDs become the actual input the model receives.
Those token IDs then get converted into embeddings: dense vectors of floating point numbers. A single token might become a 4,096-dimensional vector. These vectors encode meaning. Words with similar meanings cluster together in this high-dimensional space.
Think of it as translating each token from a flat integer into a rich coordinate in meaning-space.
This is where it gets interesting. Inference typically consists of two stages. In the prefill stage, the full input prompt is encoded in parallel to compute contextual representations for all tokens. In the decoding stage, tokens are generated one by one in an autoregressive fashion, each conditioned on previously seen tokens. Generation stops upon reaching an end-of-sequence token or a predefined length limit.
The Prefill Stage
The prefill stage corresponds to the first iteration. It processes the entire sequence of prompt tokens in parallel along the sequence dimension, passing all tokens through every decoder block. This stage achieves high compute utilization of AI accelerators due to its parallelism; however, it also incurs high latency as all prompt tokens must be processed before producing the first output token.
In plain English: the model reads your entire question at once, in one big parallel operation, and builds a rich contextual understanding of every token in relation to every other token. This is computationally intensive but fast for the hardware because GPUs are built for parallelism.
The Decode Stage
Every LLM request runs in two distinct phases: prefill, where the model reads your prompt in one parallel burst, and decode, where it generates the response one token at a time, each one depending on the last.
The decode stage is the slow part. The model cannot generate all output tokens at once because each new token depends on every token that came before it. It is inherently sequential. Token 1 is generated, then token 2 (which has seen token 1), then token 3, and so on.
In the prefill phase, the entire input prompt is processed in parallel to initialize the KV cache. This phase is dominated by dense matrix multiplications and is therefore compute-bound. In contrast, the decode phase generates tokens autoregressively, one token per step, repeatedly reading and writing the KV cache. As a result, decode execution is memory-bandwidth-bound and exposes limited parallelism per request.
What is the KV Cache?
During the decode stage, the model needs to remember the context of every token it has already seen. Recomputing all of that from scratch for every new token would be prohibitively expensive. So the model stores the Key-Value (KV) pairs from the attention mechanism in memory, and this is the KV cache. Every forward pass reads from this cache rather than recomputing.
The KV cache plays a crucial role in storing and organizing information that the model deems relevant for subsequent token generation.
The KV cache is one of the most important concepts in inference engineering. More on this when we get to optimization.
At the end of each forward pass, the model produces a vector of raw scores called logits. There is one logit for every token in the vocabulary. A vocabulary of 100,000 tokens means 100,000 raw scores.
Before any sampling happens, a language model produces raw scores called logits, one for every token in its vocabulary.
These raw scores are meaningless on their own. They can be any number, positive or negative. The next step converts them into a proper probability distribution using a mathematical function called softmax.
import torchimport torch.nn.functional as F# Example: logits for a vocabulary of 5 tokenslogits = torch.tensor([2.0, 1.0, 0.5, -1.0, -2.0])# Softmax converts raw scores into probabilities that sum to 1.0probabilities = F.softmax(logits, dim=-1)print(probabilities)# Output: tensor([0.5761, 0.2120, 0.1283, 0.0616, 0.0220])# These sum to exactly 1.0
Now the model has a probability distribution over all possible next tokens.
The question is: which token does it pick?
This is where the inference parameters come in. The model does not always pick the highest-probability token. How it samples from this distribution is controlled by a set of parameters that profoundly shape the output.
The gap between a robotic, repetitive chatbot and a creative, subtle AI assistant comes down to a single decision point: how the model picks its next token. Every time a Large Language Model generates text, it faces a vocabulary of over 100,000 candidates and must choose just one. The algorithms that make this choice are temperature scaling, Top-K, Top-P, and the increasingly dominant Min-P which determine whether the output is boringly predictable, brilliantly creative, or incoherently random.
Let’s go through each one.
Temperature is the most fundamental parameter. It controls how sharp or flat the probability distribution is.
The temperature T controls the sharpness of the distribution: as T approaches 0, the distribution concentrates on the maximum-logit token (approaching greedy decoding), while larger T produces a flatter distribution and increases randomness.
Mechanically, temperature divides all logits before the softmax is applied:
import torchimport torch.nn.functional as Flogits = torch.tensor([2.0, 1.0, 0.5, 0.2, -0.5])temperature = 0.2 # Low: more deterministicprobs_low = F.softmax(logits / temperature, dim=-1)print(f"Low temperature: {probs_low}")# Output: tensor([0.9399, 0.0536, 0.0049, 0.0014, 0.0001])# The top token dominates - very little randomnesstemperature = 1.0 # Neutral: default behaviorprobs_neutral = F.softmax(logits / temperature, dim=-1)print(f"Neutral temperature: {probs_neutral}")# Output: tensor([0.4934, 0.1815, 0.1100, 0.0822, 0.0329])temperature = 2.0 # High: more random and creativeprobs_high = F.softmax(logits / temperature, dim=-1)print(f"High temperature: {probs_high}")# Output: tensor([0.2977, 0.2022, 0.1651, 0.1494, 0.0856])# Probabilities are much more evenly spread
Practical guidelines for temperature:
Classification is deterministic by nature, you want the model’s highest-confidence prediction every time. Any randomness introduces inconsistency across repeated runs on identical tickets.
Top-K limits the model to only consider the K most probable tokens at each step. Everything outside the top K gets probability 0, and the model samples from just those K candidates.
def top_k_sampling(logits, k=5): # Get the K largest values and their indices top_k_values, top_k_indices = torch.topk(logits, k) # Create a mask — set everything outside top-k to negative infinity filtered_logits = torch.full_like(logits, float('-inf')) filtered_logits[top_k_indices] = top_k_values # Apply softmax and sample probs = F.softmax(filtered_logits, dim=-1) next_token = torch.multinomial(probs, num_samples=1) return next_tokenlogits = torch.tensor([3.0, 2.5, 1.0, 0.5, -1.0, -2.0, -3.0])token = top_k_sampling(logits, k=3)
The problem with Top-K is that K is a fixed number. Top-K imposes a hard boundary on vocabulary breadth. When the model is very confident (one token has 90% probability), K=50 still forces it to consider 49 low-probability tokens. When the model is uncertain and probabilities are spread evenly, K=50 might arbitrarily cut off many reasonable candidates.
Top-P solves exactly the problem with Top-K. Instead of a fixed count, Top-P picks the smallest set of tokens whose cumulative probability reaches a threshold P.
Top-P adapts to the model’s confidence dynamically. When the model is very confident (one token has 90% probability), Top-P=0.9 samples from just 1 to 2 tokens. When the model is uncertain (50 tokens each at 2%), Top-P=0.9 includes all 50.
def top_p_sampling(logits, p=0.9, temperature=1.0): # Apply temperature first logits = logits / temperature probs = F.softmax(logits, dim=-1) # Sort by probability (descending) sorted_probs, sorted_indices = torch.sort(probs, descending=True) # Cumulative sum cumulative_probs = torch.cumsum(sorted_probs, dim=-1) # Remove tokens once cumulative probability exceeds p sorted_indices_to_remove = cumulative_probs - sorted_probs > p sorted_probs[sorted_indices_to_remove] = 0 # Renormalize and sample sorted_probs = sorted_probs / sorted_probs.sum() next_sorted_index = torch.multinomial(sorted_probs, num_samples=1) next_token = sorted_indices[next_sorted_index] return next_token
A Top-P value of 0.9, for example, restricts selection to the smallest set of tokens whose cumulative probability reaches 90%, excluding less likely options.
LLM sampling is ultimately about working through the trade-off between coherence and creativity. Temperature changes probability. Top-K imposes a hard boundary. Top-P adapts to confidence.
Min-P is available in llama.cpp, vLLM, Ollama, and several open-source inference stacks. Adoption is growing.
Unlike Top-P’s cumulative threshold, Min-P sets a floor based on the top token’s probability. Any token with probability below (min_p × max_probability) is discarded.
The threshold scales automatically with the model’s confidence. When the model is very confident, the absolute threshold is high and few tokens survive. When the model is uncertain, the threshold is low and more tokens survive. It’s a multiplicative relationship rather than the cumulative one in Top-P, which some practitioners find easier to reason about.
def min_p_sampling(logits, min_p=0.1, temperature=1.0): logits = logits / temperature probs = F.softmax(logits, dim=-1) # Find the top token's probability max_prob = probs.max() # The threshold scales with the model's confidence threshold = min_p * max_prob # Zero out everything below the threshold filtered_probs = probs.clone() filtered_probs[probs < threshold] = 0 # Renormalize and sample filtered_probs = filtered_probs / filtered_probs.sum() next_token = torch.multinomial(filtered_probs, num_samples=1) return next_token
All these parameters do not operate independently. The default chain in
llama.cpp is:
logits → penalties → top-k → typical → top-p → min-p → temperature → sample.
The order matters enormously.
Max Tokens: Sets a hard cap on how long the response can be. If the model hits this limit mid-sentence, it stops. Always check finish_reason in your API response, if it says length, your output was cut off.
Frequency Penalty: Reduces the probability of tokens that have already appeared frequently in the output. Discourages repetitive language.
Presence Penalty: Penalizes any token that has appeared at all (not by frequency). Pushes the model toward introducing new topics and concepts.
Stop Sequences: A list of strings that, if generated, cause the model to stop immediately. Useful for structured outputs where you want the model to stop at a specific delimiter.
Not all inference is the same. The way you deploy a model, and when and where you run it, creates fundamentally different architectural patterns.
Real-time inference responds to requests as they arrive, generating a response immediately. This is what you experience every time you chat with an AI assistant.
The defining constraint is latency. Voice AI requires under 150 milliseconds total for the LLM stage. Any higher and the spoken response feels broken.
Real-time inference drives chatbots, coding assistants, search features, and any interactive AI product. The engineering challenge is serving potentially millions of simultaneous users with consistent, low latency.
Batch inference pipelines are often orchestrated using workflow management tools, with jobs executed across clusters or cloud environments for scalability. These are common for tasks such as scoring large user lists for marketing, processing image archives, or re-evaluating entire datasets for compliance or fraud checks. Unlike real-time inference, latency is less of a concern; throughput and cost-effective resource utilization are prioritized instead.
Batch inference is dramatically cheaper than real-time. You can schedule batch jobs during off-peak hours, use spot instances, and pack many requests together for maximum GPU utilization. The tradeoff is that you get the results later, not immediately.
Example use cases: nightly sentiment analysis of all customer support tickets, weekly fraud scoring of an entire user database, bulk document classification.
Edge inference brings machine learning models directly to where data is generated on devices at the network edge. This approach eliminates round-trip latency to cloud servers, reduces bandwidth costs, and enables real-time decision making in applications ranging from autonomous vehicles to industrial IoT sensors.
The challenge with edge inference is that devices have severe memory and compute constraints compared to a data center. This is why techniques like quantization (running models at lower precision) are critical for edge deployment.
Autonomous vehicles use object detection models to identify pedestrians and traffic signs in real time. Quantization enables processing 30 or more camera feeds simultaneously on vehicle hardware without cloud connectivity. Medical devices use portable diagnostic equipment that analyzes X-rays or ECG readings on-site. A quantized model runs on a tablet-sized device instead of requiring hospital servers, enabling rural clinic deployments.
Serverless inference abstracts away all infrastructure. You call an API endpoint, pay per token or per request, and someone else manages the GPUs. Services like OpenAI, Anthropic, and Google AI APIs work this way.
The tradeoff is a “cold start” penalty, the first request after a period of inactivity may take longer because the model needs to be loaded into GPU memory. Warm instances respond fast. Cold ones introduce a noticeable delay.
Here is a number that should reframe how you think about this entire field.
In late 2022, running a GPT-4-class model cost approximately $20 per million tokens. In early 2026, equivalent performance costs $0.40 per million tokens or less. That is a 1,000x reduction in just over three years, one of the fastest cost declines in computing history.
You might think: great, inference got cheap, the problem is solved. But here is the paradox.
Inference now accounts for two-thirds of all AI compute, up from one-third in 2023.
The market implication is easy to miss if you only look at model launches. Lower serving cost is not bearish for infrastructure by default. It often produces the opposite effect. Cheaper serving expands the set of economically viable products, which increases demand for GPUs, networking, storage, and memory.
Costs dropped 1,000x. Usage grew faster. Total inference spending is still climbing.
And now, inference is not just getting more frequent. It is getting more computationally demanding per request.
For years, the AI industry followed one scaling law: make training bigger. More data, more parameters, more compute during training. That was how GPT-3 became GPT-4.
In 2024, something changed. From 2020 to 2023, AI progress was dominated by training-time scaling: bigger models, more data, more compute during training. In late 2024, OpenAI released o1 and changed the paradigm.
Test-time compute is a different axis: let the model spend more compute at the moment of answering and generating a long chain of thought, exploring multiple solution paths, and verifying its own work before responding. The result trades latency and cost for accuracy, which pays off most in high-stakes domains like math, code, science, law, and medicine.
On ARC-AGI-2 the benchmark specifically designed to test genuine generalization that cannot be gamed by memorization o3 at high compute settings scored 75.7%. The previous state of the art was under 20%. By spending more compute at inference time, in the high setting o3 averages 57 million tokens per question, about 14 minutes of runtime, it achieved results that looked impossible to the research community just months earlier.
This is a fundamental shift. Inference is no longer just “run the model and return the answer.” It is increasingly “let the model think, reason, verify, and then return the answer.” That changes everything about how inference infrastructure needs to be designed.
Post-training techniques like reinforcement learning from human feedback, synthetic data augmentation, and test-time scaling, where models think through problems step by step, can use 30 to 100 times the compute of a simple inference query.
Raw inference is expensive. The engineering discipline of inference optimization is about closing the gap between what the hardware can theoretically do and what naively running a model actually achieves.
LLM inference optimization is the set of techniques that reduce the cost, latency, and memory consumption of running large language model predictions. It spans three layers: model-level (quantization, pruning, distillation), system-level (continuous batching, PagedAttention, speculative decoding), and application-level (context compression, prompt caching). Stacking optimizations across all three layers can reduce inference cost by 80% or more.
Let’s go through each layer.
Quantization
Models store their billions of parameters as floating point numbers. By default, these are stored in FP16 (16-bit) or FP32 (32-bit) format. Quantization reduces this precision.
Quantizing from FP16 to INT8 or INT4 reduces memory by 2 to 4x and cuts inference cost by roughly 50% while maintaining 95 to 99% of original accuracy. Google’s TurboQuant (2026) compresses the KV cache to 3 bits with zero measured accuracy loss, achieving 6x memory reduction.
The intuition: most of a model’s behavior is preserved even when you store numbers less precisely. A weight that is 0.73842 behaves almost identically to 0.75 for the model’s purposes.
from transformers import AutoModelForCausalLM, BitsAndBytesConfigimport torch# Load a model in 4-bit precision instead of default 16-bitquantization_config = BitsAndBytesConfig( load_in_4bit=True, bnb_4bit_compute_dtype=torch.float16, bnb_4bit_quant_type="nf4", # NormalFloat4 - better for weights bnb_4bit_use_double_quant=True # Nested quantization for memory savings)model = AutoModelForCausalLM.from_pretrained( "meta-llama/Llama-3-8B", quantization_config=quantization_config, device_map="auto")# A 16GB model now fits in ~4GB of VRAM
Knowledge Distillation
Distillation trains a smaller “student” model to mimic the behavior of a larger “teacher” model. The student learns not just from the correct answers but from the teacher’s full probability distributions a much richer training signal than just labels.
The result: a model that is 70% to 80% smaller but retains 90%+ of the teacher’s performance. DeepSeek’s approach of distilling reasoning traces from R1 into smaller models is a prominent 2026 example.
Pruning
Pruning identifies and removes parameters that contribute little to the model’s output. Some attention heads and neurons turn out to be largely redundant. Removing them reduces model size and speeds up computation with minimal accuracy loss.
Speculative Decoding
The decode stage is sequential and slow. Speculative decoding attacks this bottleneck with a clever trick.
Standard autoregressive decoding generates one token per forward pass of the large model. Each forward pass is expensive. Speculative decoding uses a small, fast “draft” model to propose multiple tokens, then the large “target” model verifies them all in a single forward pass. The key insight: verifying N tokens in one forward pass costs roughly the same as generating 1 token, because the attention computation over existing context dominates the cost regardless of how many new tokens you evaluate.
Speculative decoding accelerates inference by having a small “draft” model propose tokens that a larger “target” model verifies in parallel. Draft Phase: Small model generates K candidate tokens quickly. Verify Phase: Large model verifies all K tokens in a single forward pass. Accept/Reject: Accept matching tokens, reject and regenerate mismatches. Guarantee: Output is mathematically identical to target model alone.
A typical setup uses a 1B draft model paired with a 70B target model. The draft model proposes 4 to 8 tokens. When the draft model is right (which it frequently is for common patterns), you get 4 to 8 tokens for the compute cost of 1. Real-world speedups of 2x to 3x are common.
PagedAttention and the KV Cache
Remember the KV cache from earlier? Its management is one of the most critical aspects of serving multiple concurrent users.
PagedAttention reduced KV cache waste from 60 to 80% to under 4%.
Before PagedAttention (introduced in vLLM), serving systems had to pre-allocate memory for the maximum possible sequence length for every request. If a request only used 100 tokens but you reserved memory for 4,096, the rest was wasted. With thousands of concurrent requests, this waste became catastrophic.
PagedAttention borrows an idea from operating system virtual memory. Instead of one contiguous block per request, KV cache is stored in fixed-size “pages” that are allocated on demand. Requests share memory more efficiently, allowing far more concurrent users on the same hardware.
Continuous Batching
Traditional batching waited for a batch of requests to fill up before starting inference. This meant the first request in a batch waited for the last one to arrive. Continuous batching changes this fundamentally.
Continuous batching delivers up to 23x throughput improvement.
In continuous batching, new requests can join an in-progress batch as soon as any existing request finishes. The GPU is never idle waiting for a batch to complete. New requests slide in to replace finished ones, keeping utilization near 100% at all times.
Prompt Caching
If your system prompt is identical across many requests (which it almost always is), why recompute it every time? Prompt caching stores the KV cache for fixed prompt prefixes and reuses it across requests.
For a 4,000-token system prompt serving 10,000 requests, prompt caching eliminates 4,000 x 10,000 = 40 million tokens of redundant computation.
Context Compression
The average LLM API call wastes 40 to 60% of input tokens on context the model does not need. Stale conversation history, boilerplate system prompts, full-file includes when only three functions matter. You pay for every wasted token twice: once in your API bill, again in latency while the model attends over padding.
Context compression removes irrelevant tokens from the input before sending them to the model. Summarizing old conversation history instead of appending it verbatim is one form. Retrieving only relevant chunks from a document (RAG) instead of sending the whole document is another.
Model Routing
Not every request needs a 70B parameter frontier model. Simple questions can be answered just as well by a 7B model at a fraction of the cost. Routing systems classify incoming requests by complexity and send them to appropriately sized models.
Five primary strategies reduce inference costs: quantization (2 to 4x savings), response caching for repeated queries (3 to 10x savings), prompt optimization to reduce token usage (1.5 to 3x savings), model routing to use smaller models for simple tasks (2 to 5x savings), and batch processing for throughput-oriented workloads (1.3 to 2x savings). These techniques compound when combined effectively.
You cannot optimize what you cannot measure. These are the five metrics every inference engineer tracks.
Time to First Token (TTFT)
The time to first token (TTFT) is essentially the latency of the prefill stage. Prefill often takes significantly longer than any single decode step because it may involve processing hundreds of input tokens to produce that first output token.
TTFT determines how long the user waits before any response appears. For interactive applications, this is the most user-visible metric. High TTFT makes the system feel unresponsive even if the eventual throughput is fast.
Time per Output Token (TPOT)
For the decode stage, Tokens-Per-Second (TPS) or Time-Per-Output-Token (TPOT) is used to measure the generation rate of LLM serving.
TPOT is the cadence of the streaming response. If TPOT is too high, the text appears to trickle out uncomfortably slowly. Human reading speed is roughly 250 words per minute, or about 300 tokens per minute. TPOT needs to be at most 200ms to match human reading speed and ideally much faster to feel fluid.
Throughput (Tokens per Second)
Total tokens generated per second across all concurrent requests. This is the primary metric for batch workloads and for measuring infrastructure efficiency. Higher throughput means lower cost per request.
Time to Last Token (TTLT) / End-to-End Latency
The total wall-clock time from request submission to complete response. For batch workloads and automated pipelines, this is often more important than the streaming metrics.
GPU Utilization
A fast P99 at 40% GPU utilization means you have headroom. A fast P99 at 98% means you are one traffic spike from degradation.
High utilization is good for throughput but leaves no buffer for traffic spikes. The practical target for production serving is 70% to 85% utilization.
Three tiers of inference hardware have emerged, each with a different cost and capability profile.
The gold standard for frontier model inference. Cloud H100 GPU pricing has stabilized at $2.85 to $3.50 per hour across major providers as of early 2026.
Cloud inference handles frontier models, multi-tenant workloads, and anything requiring FP16 or FP8 precision at scale. No upfront cost. Linear scaling. Pay-as-you-go.
The tradeoff: at very high sustained volumes, cloud GPU becomes more expensive than self-hosting. For most teams in 2026, API inference is cheaper than self-hosting once engineering labor is fully accounted for. Self-hosting makes financial sense at sustained production volumes exceeding 5 to 10 million tokens per month against premium APIs.
The RTX 5090 (32GB VRAM) and RTX 4090 (24GB VRAM) can run quantized models locally. A 7B Q4 model runs at 150 to 260 tokens per second on the RTX 5090.
For developers running local inference, Apple Silicon has also become a strong option. M3 and M4 chips have fast unified memory that can run quantized 7B and 13B models at usable speeds.
Smartphones (via Apple’s Core ML and Android’s NNAPI), IoT sensors, Raspberry Pi, Jetson modules. These run the smallest, most aggressively quantized models (1B to 3B parameters). Latency is sub-100ms for short inputs. Privacy is inherent, no data leaves the device.
In 2026, sophisticated products do not pick one tier. They use all three.
Tier 1 (Edge) handles classification, simple question-and-answer, short completions, and all latency-sensitive tasks. Cloud GPU handles frontier model inference for complex tasks. A routing layer decides which tier each request goes to based on complexity, latency requirements, and cost.
Split inference divides model execution between the edge and the cloud. Early layers process locally for speed and privacy.
Even with all the optimization techniques above, inference at scale has real unsolved problems.
The Cold Start Problem
Serverless inference environments spin down GPU instances when idle to save cost. The first request after idle spins up a new instance and loads the model weights into VRAM. For a 70B parameter model in FP16, that is 140GB of data to transfer. Cold starts can take 30 to 60 seconds, which is catastrophic for user-facing products.
Solutions include keeping “warm” instances alive at baseline cost, or using model weight caching at the infrastructure level.
KV Cache Memory Pressure
As context windows grow (Claude supports 200K tokens, Gemini 1 million), the KV cache per request becomes enormous. A single long-context request can use as much GPU memory as hundreds of normal requests.
The ultimate constraint in scaling model serving is not raw compute power but memory bandwidth and the strict management of the KV cache.
Hallucination at High Temperature
At high temperature settings, the model explores unlikely parts of the probability distribution. Sometimes this produces creative, valuable outputs. Sometimes it produces confident falsehoods. There is no temperature value that eliminates hallucination while preserving creativity it is a fundamental tradeoff in probabilistic sampling.
Interference Between Prefill and Decode
When prefill and decode are co-located on the same GPUs, prefill requests can monopolize GPU execution and delay decode steps, directly inflating both TTFT and TPOT.
This is why “prefill-decode disaggregation” running the two phases on separate hardware has become an active research and engineering direction.
Agentic Cost Explosion
Agentic workflows that make 50 to 200 LLM calls per task turn a cheap per-token price into an expensive per-task cost.
A single user action in an AI agent might trigger 100 inference calls. At $0.40 per million tokens and 1,000 tokens per call, that is $0.04 per user action, orders of magnitude higher than traditional software. Building economically sustainable agents requires aggressive optimization at every layer.
80% of AI GPU spend is now inference. The industry has crossed the inflection point. Training is a line item. Inference is the operating model.
The inference chip wars became one of the most important infrastructure stories of the first half of 2026. AI deployment has moved from training to serving billions of tokens and the competition is about cost per token, latency, power efficiency, and context handling. Inference is fragmenting by workload, creating room for specialized hardware beyond GPUs.
Custom ASICs (application-specific integrated circuits) designed purely for inference and not training and are entering production. They trade general-purpose flexibility for dramatically better cost-per-token on specific workload types.
Architectures are getting more efficient, often using sparse MoE designs so only a small part of the model is active per token. Qwen3-Coder-Next is one example, with an ultra-sparse setup and a 256k native context window.
MoE models have a total parameter count that looks large on paper but only activate a fraction of parameters per token. A model with 100B total parameters might only use 20B for any given token. This gives frontier-model quality at a fraction of the inference cost.
Analysts project that by 2030, inference compute will represent 75% of total AI compute, a complete inversion from the training-dominated paradigm of the early 2020s.
The AI inference market grows from $106 billion (2025) to $255 billion (2030) at 19.2% compound annual growth rate.
Reasoning models that think longer at inference time o3, DeepSeek-R1, Claude with extended thinking are redefining what “a single inference call” means. It no longer means one forward pass. It means dozens to thousands of forward passes organized into chains of thought, self-verification loops, and multi-step reasoning.
Advances in distillation, quantization, and memory-efficient runtimes are pushing inference to edge clusters and embedded devices, driven by cost, latency, and data-sovereignty needs.
The combination of smaller, more capable models (3B to 7B parameter models that rival 2022-era frontier performance) and improved quantization means serious AI capabilities are now deployable on consumer hardware with no cloud dependency.
If you are a developer or engineer, inference is becoming as fundamental as databases or APIs in modern AI application development. Knowing how it works helps you design faster, cheaper, and more reliable systems. Poor inference implementation can lead to slow response time, high compute costs, and a poor user experience.
Here is what you should walk away with:
Inference is the process of running a trained, frozen model on new inputs. The weights do not change. The model applies what it learned during training.
Every response is generated one token at a time, through a two-phase process of prefill (reading your input in parallel) and decode (generating output sequentially).
The parameters: temperature, Top-P, Top-K, Min-P are directly shape the creativity, coherence, and determinism of every response.
Optimization across three layers (model, system, application) can reduce costs by 80% or more without sacrificing quality.
In 2026, inference is not just a serving problem, it is a product problem. Cost per token determines which products are viable. Latency determines which experiences feel good. The engineers who understand inference deeply are the ones building the products that survive.
The next time you type a prompt and watch the response stream in token by token, you now know exactly what is happening inside the machine.
Q1. Does the model learn or update itself when I send it a message?
No. During inference the model’s weights are completely frozen. Every parameter are all the billions of numbers that encode the model’s knowledge was locked in place when training ended. Your message travels through the model’s layers, gets processed, and a response is generated, but nothing inside the model changes as a result. If you send the exact same question a thousand times, the weights stay identical each time. The only thing that can change the weights is a new training or fine-tuning run, which is a separate, deliberate process done by engineers, not by users chatting with the model.
Q2. Why does the same prompt sometimes give different answers?
Because of sampling. The model does not deterministically output one fixed answer. At each step it produces a probability distribution over all possible next tokens, and a sampling algorithm picks from that distribution. Parameters like temperature, Top-P, and Top-K control how that sampling works. When temperature is above 0, there is inherent randomness in the process. Even with an identical prompt and identical model weights, the roll of the dice at each token step can diverge the output in different directions. Setting temperature to 0 (greedy decoding) makes the output deterministic, but most production systems keep some temperature to avoid robotic, repetitive responses.
Q3. What is the difference between a parameter and a token?
These two words get confused constantly so it is worth being precise. A parameter is a number stored inside the model, a weight that was learned during training. A 70 billion parameter model has 70 billion of these internal numbers baked into it. They never change during inference. A token, on the other hand, is a chunk of your input or output text, roughly a word or subword. Tokens are not stored in the model; they are the data that flows through it. Every request generates new tokens. The model’s parameters process those tokens but exist completely independently of them. Parameters live in the model. Tokens live in the conversation.
Q4. Why is inference so much more expensive than just running a regular program?
Three reasons. First, even a small model has billions of floating-point multiplications happening per forward pass, which requires GPU hardware that costs thousands of dollars per card. Second, the autoregressive decode stage is memory-bandwidth-bound rather than compute-bound, meaning the bottleneck is how fast the GPU can read its own memory, not how fast it can do math. Third, running at scale means serving thousands of concurrent users simultaneously, each with their own context window, their own KV cache consuming GPU memory, and their own latency requirements. Unlike a traditional API that runs a few database queries, LLM inference requires holding enormous model weights in expensive high-bandwidth memory and re-reading them for every single token generated.
Q5. What actually happens during a “cold start” and why does it feel slow?
A cold start happens when a serverless inference instance has been idle long enough that the provider spun it down to save cost. When your request arrives, the system first has to spin up a new GPU instance, then load the model weights from storage into GPU VRAM. For a 7B model in FP16 that is 14GB of data to transfer, and for a 70B model it is 140GB. This transfer takes time often 30 to 60 seconds depending on the model size and the storage infrastructure. After the weights are loaded, the instance is “warm” and subsequent requests on it respond in milliseconds. The fix for production systems is keeping a minimum number of warm instances alive at all times at baseline cost, accepting a small idle charge to eliminate cold start latency entirely.
Q6. Can a model run inference and training at the same time?
Technically they are different compute operations and can run on separate hardware in parallel, but a single model instance cannot do both simultaneously. During inference the weights are read-only and used to generate outputs. During training, gradients flow backward and weights are updated. Running both operations on the same weights at the same time would produce undefined behavior. In practice, companies run the serving (inference) version of a model on one set of hardware and train updated versions on separate hardware. When a new version is ready it gets deployed to replace the previous one, and the serving infrastructure switches over sometimes gradually using a technique called canary deployment.
Q7. Is a larger model always better for inference?
Not always, and in many real scenarios a smaller model is the better choice. A 7B parameter model running at INT4 precision can answer simple questions just as accurately as a 70B model, but at a fraction of the cost and latency. Larger models excel at complex multi-step reasoning, nuanced writing, and hard coding problems. For classification, summarization of short documents, simple Q&A, and routing tasks, smaller models are faster, cheaper, and perfectly accurate. The most efficient production systems use model routing and sending easy requests to small models and hard requests to large ones, rather than running every query through the biggest available model.
Q8. What is the KV cache and why does everyone keep talking about it?
KV stands for Key-Value, which comes from the attention mechanism inside the transformer architecture. During the decode stage, every new token needs to attend to every previous token in the conversation. Without caching, the model would recompute the key and value vectors for all previous tokens from scratch at every single step an exponentially growing amount of redundant work. The KV cache stores those computed vectors in GPU memory so each new decode step only has to compute the new token’s keys and values and look up the rest from cache. This makes autoregressive generation tractable. The downside is that the KV cache consumes a large and growing amount of GPU memory as the conversation gets longer, which is why long-context windows are expensive and why systems like PagedAttention were invented to manage that memory efficiently.
These are the sources that informed the research and claims in this blog. Each is worth reading if you want to go deeper on a specific topic.
1. “Efficient Memory Management for Large Language Model Serving with PagedAttention” — Kwon et al., 2023
https://arxiv.org/abs/2309.06180
2. “A Survey of LLM Inference Systems” — Pan et al., arXiv, June 2025
https://arxiv.org/abs/2506.21901
3. vLLM Official Documentation
4. “Inside Real-Time LLM Inference: From Prefill to Decode, Explained” — Dev Patel, Medium, 2025
5. “What is Inference Engineering?” — Gergely Orosz, The Pragmatic Engineer, March 2026
https://newsletter.pragmaticengineer.com/p/what-is-inference-engineering
6. “AI Inference vs Training: Key Differences Explained” — DigitalOcean, 2026
https://www.digitalocean.com/resources/articles/ai-inference-vs-training
If this guide helped you, follow for more deep-dives into AI infrastructure, LLM engineering, and the practical mechanics of building with language models.
The Complete Guide to Model Inference: Every Time You Type a Prompt, This Is What Happens was originally published in Towards AI on Medium, where people are continuing the conversation by highlighting and responding to this story.