TL; DR
The whole stack exists to answer one question: where does a request’s state live, and what does it cost?
Capacity is set by two independent ceilings — how many requests fit in memory, and how many the latency target allows. The lower one binds.
GPU utilisation is a misleading metric. Judge a deployment on goodput: requests completed within your latency targets.
Most production incidents are diagnosable from three symptoms — bad TTFT, bad TPOT, or stalling generations — each pointing at a different layer.
Cost per million tokens is driven overwhelmingly by concurrency and cache hit rate, not by hardware choice.
Build this in stages. Every layer should be forced by a measured limit, never adopted preemptively — and for many teams, a hosted API remains the right answer.
Someone types a question and presses send. A second later, words start appearing.
Between those two moments sits a system with more layers than almost anything else in modern infrastructure: an API gateway, a router that understands token economics, an orchestrator managing GPU nodes, an inference engine juggling hundreds of concurrent sequences, a memory manager allocating attention state in pages, and a model whose parameters may be spread across a dozen accelerators.
This article does two things. First it lays out that stack quickly enough to be useful on its own — no prior reading assumed. Then it deals with the part that only matters once the architecture is settled: how you actually run it. Sizing it, measuring it, diagnosing it at three in the morning, paying for it, and deciding which pieces you need at all.
Five layers, each solving a problem created by the one below.
A large language model is an architecture plus a very large set of learned parameters called weights. Memory needed is roughly parameters × bytes per parameter, so a 70-billion-parameter model in 16-bit precision needs about 140 GB — before anything else.
That memory has to be VRAM, the high-bandwidth memory attached to the GPU itself. System RAM is not a substitute; having 512 GB of it does not help an 80 GB accelerator. Four resources govern everything that follows: compute capacity, memory capacity, memory bandwidth (how fast data moves between VRAM and the compute units), and network bandwidth (how fast GPUs and machines exchange data).
When weights exceed one GPU, there are four responses. Quantization stores parameters in fewer bits — 8-bit roughly halves the footprint against 16-bit, 4-bit halves it again, with some accuracy cost. Off keeps part of the model in CPU memory, which solves capacity at a steep bandwidth price. Sharding splits one model across several GPUs. Replication makes complete copies.
The last two are constantly confused and solve opposite problems: sharding addresses model capacity, replication addresses workload capacity. Sharding comes in flavours — tensor parallelism splits the operations inside each layer and demands a fast interconnect, so it suits GPUs within one machine; pipeline parallelism gives each GPU a contiguous group of layers and communicates far less, so it tolerates spanning machines.
A request runs in two phases with opposite hardware appetites.
Prefill processes the entire prompt at once. All prompt tokens are known, so the work is highly parallel — prefill is compute-bound.
Decode generates the answer one token at a time, each token depending on all those before it. To produce a single token, the GPU must read the model’s entire weight set from memory. Decode is memory-bandwidth-bound, and the compute units sit largely idle waiting.
That last fact is the basis of the entire serving economy. If reading 140 GB of weights yields one token, it is wasteful — so the engine batches, processing the next token for many sequences in one pass. The same weight read now produces dozens of tokens. Modern engines use continuous batching, rebuilding the batch before every forward pass so finished requests leave and waiting ones join immediately.
Crucially, the weights are read-only and shared by everyone. What is per-request is the KV cache.
Attention requires, for each new token, the keys and values of every preceding token. Those never change once computed, so the engine stores them rather than recomputing. That store is the KV cache, and it is the dominant dynamic consumer of GPU memory.
Its size is fixed per token by the architecture and grows along two axes: context length and concurrency. For a large model it can run a few hundred kilobytes per token — meaning a hundred users with 8,000-token conversations can consume more memory than the model weights themselves.
Because sequence lengths are unpredictable, engines allocate it in small fixed-size blocks on demand rather than reserving a maximum-length region per request. That technique — PagedAttention — eliminates most of the waste and is where a large share of modern serving throughput comes from. When the pool fills, running requests get preempted.
The consequence people underestimate: concurrency is capped by cache memory, not compute. “How many users can this GPU serve?” is unanswerable without a context length attached.
Production traffic is repetitive. Every request carries the same system prompt; every chat turn resends the whole conversation; many users query the same document.
Attention is causal, so a token’s state depends only on what precedes it — meaning two requests beginning with identical tokens produce an identical cache for that shared portion. Prefix caching retains it so the second request skips that prefill entirely. Engines identify reusable blocks by hashing each one together with the hash of everything before it, so a match implies the entire preceding sequence matched.
This is why variable content at the front of a prompt is expensive: a timestamp at the top changes the first block’s hash, which changes every hash after it, destroying every match.
Once there are many replicas, classic load balancing fails. Its assumptions — that requests cost roughly the same, are short, and that servers are interchangeable — are all false here. One request can cost a hundred times another, occupy memory for a minute, and be nearly free on the one replica holding its cached prefix. An LLM-aware router scores candidates on cache locality, cache utilisation and queue depth measured in tokens.
Kubernetes runs the fleet: placing servers on nodes with the right accelerators, replacing failures, gating traffic on readiness probes, scaling replicas. It never touches a token. Its blind spot is that it models an inference server as a generic container with GPUs attached.
Inference-aware platforms such as llm-d close that gap, making a pool of model servers a first-class object with pluggable, state-aware scheduling — and going further by running prefill and decode on separate GPU pools, transferring cached state between them, and letting that state spill beyond one GPU into host memory, disk or shared cluster storage.
Everything above is architecture. What follows is the part that decides whether it works.
The first question anyone asks, and it has a real answer — with a catch.
Work down from total memory:
Total VRAM 640 GB (8 × 80 GB)− model weights (70B, FP16) −140 GB− runtime, buffers, overhead − 40 GB ───────= KV cache pool 460 GB
Then divide by what one request consumes. At roughly 320 KB per token for this model class, a request averaging 4,000 tokens of context needs about 1.3 GB:
460 GB ÷ 1.3 GB ≈ 350 concurrent requests
That is your memory ceiling. Now the catch: it is not the only ceiling.
Concurrency also has a latency ceiling. Larger batches raise throughput but slow every individual response, so if your TPOT target is 50 ms per token, there is a batch size beyond which you cannot meet it — regardless of available memory. That number comes from benchmarking, not arithmetic, because it depends on the model, the hardware and the request mix.
Your real capacity is the lower of the two. Which one binds tells you what to do about it: memory-capped means quantize, shard, or shorten context limits; latency-capped means add replicas, because more memory on the same GPU will not help.
Then subtract headroom. A deployment sized exactly to peak has no room for a traffic spike, and inference replicas take minutes to start — image pull, weight , engine initialisation. You cannot scale into a spike reactively, so you either hold spare capacity or accept degradation.
Three numbers describe user experience, and a single average conceals all of them.
TTFT — time to first token. How long before anything appears. Driven by queueing and prefill. This determines whether the interface feels alive.
TPOT — time per output token. How fast the answer reads out once started. Driven by decode. Comfortable reading is roughly 30–50 ms per token; below that, further gains go unnoticed.
ITL — inter-token latency. The gap between individual streamed tokens. Related to TPOT but exposes variance — the stutters caused by other requests’ prefill work, which an average hides completely.
The number that ties them together is goodput: requests completed while meeting your targets. A server reporting excellent token throughput while every user waits eight seconds for a first token has superb throughput and terrible goodput.
Throughput measures what the hardware did. Goodput measures what the users got. Optimise the second.
Set targets by workload, not by aspiration. Interactive chat needs fast TTFT and can tolerate moderate TPOT. Batch document processing barely cares about either and should be tuned purely for throughput. Voice needs both to be tight. Running these on shared infrastructure with one configuration means serving all of them badly.
Start with what does not: GPU utilisation.
That figure reports whether any kernel was executing, not whether the compute units did anything useful. A GPU in a decode loop, stalled on memory reads with most of its arithmetic idle, reports near-100% happily. High utilisation with low token throughput is not a healthy server — it is a bandwidth-starved one, and the fix is a bigger batch, not a bigger GPU.
What to graph instead:
KV cache utilisation, per replica. The best single indicator of how close a server is to trouble, because it is the resource that actually runs out. It is also your load-balance check — a wide spread across replicas means the router is not seeing real load.
Queue depth in tokens, not requests. Five queued requests of 200 tokens is nothing; five of 30,000 is a wall everyone behind will wait through.
Preemption rate. Non-zero means the cache pool is exhausted and running requests are being evicted. This is the clearest signal that you are past capacity.
Prefix cache hit rate. If it is far below what a single server achieves, routing is destroying locality. If it is near zero on repetitive traffic, look for variable content at the front of your prompts.
TTFT and TPOT at p95 and p99, never the mean. Failures live in the tail. A router that is right 90% of the time and catastrophic otherwise looks fine on average.
Tokens per second, input and output separately. They represent different work and different costs.
You cannot size a deployment from a spec sheet, because capacity depends on your request shapes.
Benchmark with your distribution of prompt and output lengths. A model benchmarked at 512-in/128-out will behave nothing like the same model serving 20,000-in/500-out, and the published number will mislead you badly.
Sweep request rate upward and watch where latency breaks. The useful output is not a single throughput figure but a curve: goodput against load, with the knee marked. That knee is your operating point, and the gap between it and your provisioned capacity is your headroom.
Two mistakes to avoid. Benchmarking with an unrealistically high cache hit rate — sending the same prompt repeatedly gives you numbers you will never see in production. And benchmarking at fixed concurrency rather than fixed arrival rate, which hides queueing entirely, since queueing is exactly what happens when arrivals exceed service.
Most incidents present as one of a few symptoms, and each points at a specific layer.
TTFT is high, TPOT is fine. The problem is before generation: queueing, admission or prefill. Check queue depth in tokens, then prefix cache hit rate — a dropped hit rate turns cheap requests into expensive ones instantly. Then check load balance; one saturated replica while others idle is a routing problem, not a capacity problem.
TPOT is high, or output stutters. Generation itself is slow. Either the batch is too large — throughput was traded for latency — or large prefills are interrupting decode. Chunked prefill smooths the second; reducing concurrency addresses the first.
Requests stall mid-generation or fail under load. Cache exhaustion, almost certainly. Check preemption rate and cache utilisation. Fixes are to reduce concurrency, cap maximum context, quantize the KV cache, or add capacity.
Throughput is low but GPU utilisation is high. Bandwidth-bound with too small a batch. Raise concurrency limits. This is the classic misread that leads teams to buy more GPUs they do not need.
One replica is hot, others idle. Routing. Either it is counting connections instead of measuring load, or session affinity has concentrated traffic on one server.
Everything degraded right after a deploy or scale-up. Cold caches. New replicas start with nothing, and warm up over minutes. Expected, temporary, and worth confirming rather than chasing.
Pods stuck Pending. GPU fragmentation. A Pod requesting eight GPUs needs eight free on one node; sixteen scattered two-per-node across eight machines will not do.
GPUs are the dominant cost, and the metric that matters is cost per million tokens:
GPU cost per hourcost per M tokens = ─────────────────────── × 1,000,000 tokens produced per hour
The numerator is largely fixed once you have chosen hardware. Almost all the leverage is in the denominator — which means cost optimisation is throughput optimisation.
In rough order of impact:
Concurrency. The single largest lever. Running at batch sizes near your latency limit rather than well below it can change the number several-fold, because the expensive weight read is amortised across everything in the batch. A GPU serving 20 concurrent requests when it could serve 200 costs roughly ten times more per token.
Prefix cache hit rate. In workloads with large shared prompts, hits eliminate most of the prefill work. Restructuring prompts so stable content sits at the front is close to free and can be worth a great deal.
Model size and precision. A smaller or quantized model produces more tokens per GPU-hour. The real question is whether quality holds for your task — often it does, and the saving is large.
Idle provisioned capacity. Headroom held for spikes is paid for continuously. This is a real and unavoidable cost, but it should be a deliberate number, not an accident.
Context length. Every token of context consumes cache memory for the life of the request, reducing how many requests fit. Trimming conversation history is a direct capacity gain.
Note what is not on this list: choosing a marginally faster accelerator. Hardware matters, but a well-tuned deployment on modest hardware routinely beats a poorly-tuned one on better hardware, because the difference between batch-of-20 and batch-of-200 is larger than the difference between GPU generations.
Worth asking before the adoption path, because for many teams the honest answer is no.
Hosted APIs remove every problem in this article. No GPUs to provision, no cache to tune, no Pods stuck Pending at midnight. You pay per token and someone else runs the platform. For most applications, at most volumes, that is the correct choice — and the engineering time saved is usually worth more than the unit-price difference.
Self-hosting starts to make sense on a few specific grounds. Volume, once you are large enough that per-token pricing exceeds the fully-loaded cost of running your own hardware, including the people who operate it. Data control, where requests cannot leave your infrastructure for regulatory or contractual reasons. Model control, where you need a fine-tuned or open-weight model that no provider hosts. Predictable, shaped traffic, where you can keep GPUs busy — idle accelerators are the fastest way to make self-hosting more expensive than an API. And latency, where you need the model physically close to something.
The trap is underestimating the operational cost. GPUs are the visible expense; the engineering time to tune, monitor, upgrade and debug this stack is the one that gets left out of the comparison and then dominates it.
Self-host because you have a reason that survives arithmetic, not because it feels like the serious choice.
The stack described in Part 1 is what a large deployment looks like after years of pressure. Building it upfront is a mistake. Each layer is complexity, and complexity is only worth paying for when a measured limit forces it.
A reasonable order:
Start with one model server on one GPU. Enable prefix caching. Measure your actual prompt and output distributions — most teams are surprised by them.
Tune before you scale. Adjust memory allocation and concurrency limits until you reach either the memory or latency ceiling. Many deployments stop here and should.
If the model does not fit, make it fit. Quantize first — it is cheaper than more hardware and often costs nothing in practice. Shard only if it still does not fit.
If one server is not enough, add replicas behind a simple load balancer. Confirm the fleet-wide cache hit rate has not collapsed; if it has, that is your signal to add cache-aware routing.
Add Kubernetes when operating the fleet by hand becomes the bottleneck — when you have enough replicas that failures, rollouts and placement need automating.
Add inference-aware orchestration when the workarounds hurt more than the platform would. Cache-aware routing when locality is being wasted. Disaggregation when prefill interference is visibly damaging inter-token latency and you have the interconnect to support moving cache between machines.
Every layer in this stack should be forced by a number you measured, not adopted because it appeared in an architecture diagram.
Sizing on model weights alone. The most common error in the field. A 140 GB model does not fit comfortably on 160 GB of VRAM, because concurrent requests need cache and there will be almost none left.
Optimising the average. Users experience the tail. A median TTFT of 200 ms with a p99 of nine seconds is a bad experience for a meaningful number of people every hour.
Treating GPU utilisation as the health metric. Covered above, and worth repeating because it drives real purchasing decisions.
Injecting variable content at the top of prompts. A session ID above the system prompt can cost you the entire prefix cache.
Aggressive scale-down. Removing a replica discards a warm cache that costs minutes to rebuild. Scale down slowly and conservatively.
One configuration for mixed workloads. Interactive and batch traffic want opposite tuning. Separate pools serve both well; one shared pool serves both badly.
Liveness probes that fire during startup. An inference Pod takes minutes to load weights. A short liveness timeout kills it mid-load, forever, and looks like a broken image rather than a misconfigured probe.
Every layer of this stack is an answer to the same question: where does a request’s state live, and what does it cost?
Model weights are state that must fit in memory — which gives us quantization and sharding. Attention state is per-request state that must be stored rather than recomputed — which gives us the KV cache, and paged allocation to store it efficiently. That state turns out to be shareable between requests — which gives us prefix caching. Being shareable makes its location valuable — which gives us cache-aware routing. Being located on a specific Pod makes it fragile — which is why a restart is expensive in a way orchestrators do not model. And making it survive, move and tier across a cluster is precisely what inference-aware platforms are for.
Compute is rarely the limit. Memory, and the movement of state through it, almost always is.
That is also the practical takeaway. When a deployment misbehaves, the productive question is not “do we need more GPUs?” It is: which resource is exhausted, and where is state going that it should not? Is the cache pool full? Is a cache being recomputed that already existed somewhere? Is a phase of the request interfering with another? Nearly every answer in this series is a specific instance of that question.
The tools will change. Engines will get faster, orchestration will absorb more of what is bolted on today, and some of the specific projects named here will be superseded. The four resources will not change, and neither will the fact that a request’s cost is determined by its state.
You now have the map. The rest is measurement.
1. What really happens when you click ‘Send’ on ChatGPT — A journey through modern AI Infrastructure
2. What Do You Do With a Model That’s Too Big for Your GPU? — Quantization, Sharding and Parallelism Explained
3. How Does One GPU Serve Hundreds of Users at the Same Time? — Inside an LLM inference server
4. The KV Cache Explained: Why Long Conversations Get Expensive — How LLMs remember context without recomputing everything
5. Why Is Your LLM Recomputing the Same Prompt 1,000 Times a Day? — Prefix caching, radix trees and block hashing explained
6. Why Traditional Load Balancing Breaks for LLMs — Building an LLM-aware router
7. Kubernetes for LLM Inference: How AI Workloads Run Across a GPU Cluster
8. LLM-D Explained — How modern AI infrastructure routes, schedules and scales LLM inference
9. Inside a Modern AI Inference Platform — The full stack end-to-end
Sources
Inside a Modern AI Inference Platform was originally published in Towards AI on Medium, where people are continuing the conversation by highlighting and responding to this story.