TL; DR
Classic load balancing assumes requests are roughly equal, short-lived, and that servers are interchangeable. For LLM inference, all three are false.
Two requests can differ by more than a hundred times in compute and memory cost, and one may occupy a server for a minute while the other finishes in two seconds.
Counting connections tells you nothing useful. Ten short requests can be a lighter load than two long ones.
The signals that matter are KV cache utilisation, queue depth in tokens, and cache locality — where the state for this request already lives.
Locality and load pull in opposite directions, so a router has to score them against each other rather than optimise either alone.
Output length is unknown at routing time, which makes every routing decision a decision under uncertainty.
A request’s two phases want opposite things from a server, so one placement decision has to serve two incompatible workloads.
You have a working inference server. It batches efficiently, manages its memory carefully, and caches prompt prefixes so repeated work is skipped. Traffic grows past what one machine can handle, so you run twenty replicas and put a load balancer in front.
And performance gets strange. Some requests are fast, others inexplicably slow. One replica is pinned while another idles. The prefix cache hit rate, which was excellent on a single server, has collapsed.
Nothing is broken. The load balancer is doing exactly what it was designed to do. The problem is what it was designed for.
Load balancing is a solved problem for ordinary web services, and it is solved because ordinary web traffic has convenient properties.
Requests cost roughly the same. One API call is much like another. Variation exists, but it is bounded — no request is a hundred times more expensive than its neighbour.
Requests are short. Milliseconds to a second or two. A bad placement decision is corrected almost immediately by the next request.
Servers are interchangeable. Any server can handle any request equally well. State lives in a database, not in the server.
Load is observable from outside. Because requests are similar and brief, the number of open connections is a decent proxy for how busy a server is.
Every strategy in the standard toolkit — round robin, least connections, weighted variants — is built on these four. And LLM inference violates all four.
Consider two requests arriving at the same moment.
Request A 200 input tokens → 50 output tokensRequest B 30,000 input tokens → 2,000 output tokens
To the load balancer these are identical: one HTTP request each. In reality they are not close to comparable.
Prefill. Request B has 150 times more input to process. That work is real GPU compute and it happens before either request produces a single token.
Memory. The KV cache — the attention state the server must hold for the whole life of a request — scales with token count. At a few hundred kilobytes per token for a large model, A occupies tens of megabytes. B occupies several gigabytes. On a server with limited spare VRAM, B might consume a meaningful fraction of the entire pool while A is a rounding error.
Duration. A finishes 50 decode steps and releases everything, likely within a couple of seconds. B runs 2,000 decode steps, holding its multi-gigabyte cache the entire time — possibly a minute or more.
So one request is briefly negligible, and the other occupies a large share of a server’s memory for a long time. A round-robin balancer alternates between them as if they were the same thing, and load diverges within seconds of starting.
A normal request is over before a bad decision matters. An LLM request is not.
If you route a 4,000-token generation to a server that is already saturated, that request will be slow for its entire lifetime — potentially a minute — and it will make everything already running on that server slower for the same period, because they now share memory and batch slots with it.
Routing mistakes in LLM serving are not transient. They persist, and they compound.
This is the assumption that breaks most interestingly, and it follows directly from prefix caching.
Inference servers keep the computed attention state for prompt prefixes they have already seen, so that a later request beginning with the same tokens can skip that work. This means each replica accumulates its own cache, containing whatever it happens to have processed. Those caches are not shared between replicas.
So the replicas are no longer interchangeable. A request whose prefix is already cached on Server A can skip most of its prefill there and answer almost immediately. Sent to Server B, the same request pays the full cost.
The consequence is counter-intuitive enough to state plainly:
The best destination is not necessarily the server with the most spare capacity. It may be the server that already holds the right state.
There is also a compounding effect that explains the collapsed hit rate from the opening. Round robin spreads requests evenly, which means the same system prompt gets prefilled and cached on every replica independently. Twenty replicas hold twenty copies of the same prefix, each consuming memory, and every replica’s cache is churning through unrelated traffic. Even distribution actively destroys locality.
Least-connections routing sends each request to whichever server has the fewest open connections. It fails here for the same reason round robin does — a connection is not a unit of work.
Server A 10 requests short contexts 20% of cache pool usedServer B 2 requests very long contexts 85% of cache pool used
Least connections sends the next request to Server B, which has one fifth the connections and four times the memory pressure. It is the worst available choice, and the algorithm cannot see it.
The general problem: every classic strategy measures how many things a server is doing. What matters for an inference server is how much memory those things are consuming, and for how much longer. Those are different quantities, and the first is not a proxy for the second.
If counting connections does not work, what does? An LLM-aware router bases decisions on the state of the inference servers themselves. Three signals do most of the work.
The single most useful number a router can have.
An inference server’s real capacity limit is memory. Model weights occupy a fixed portion of VRAM; what remains is the pool from which every active request draws attention state. When that pool fills, the server stops admitting new requests — and starts evicting running ones, which appears to users as responses stalling mid-generation.
So cache utilisation is a direct measure of how close a server is to trouble, in a way that connection counts and even GPU utilisation are not. A server at 90% cache occupancy should receive very little new work regardless of how few connections it holds.
Requests waiting to be admitted matter, but how many are waiting is the wrong unit. Five queued requests carrying 200 tokens each is a trivial backlog. Five carrying 30,000 tokens each is a substantial amount of prefill that everyone behind it will wait through.
Measuring the queue in tokens rather than requests gives an estimate of how long a new arrival would actually wait — which is what TTFT depends on.
The value signal from Section 4: does this server already hold cached state for this request’s prefix?
Determining this cheaply is the practical challenge. A router cannot inspect every replica’s cache on every request. The common approaches are to have servers report which prefixes they hold, or to hash the request’s prefix and route consistently, so identical prefixes deterministically land on the same replica and locality emerges without the router tracking anything.
Two further signals matter in larger deployments: model identity, since a replica can only serve requests for a model it has loaded, and health and readiness, since a server still weights into VRAM must receive nothing at all.
Here is where LLM routing stops being a matter of collecting better metrics.
The two most valuable signals disagree. Cache locality says: send this request to the server that already has its prefix. Load says: send it to the server with free memory. When one server holds the hot prefix, those are frequently different servers.
Following locality alone is a trap. If a single system prompt dominates your traffic, every request has the same locality preference, and pure locality routing herds all of them onto one replica while nineteen sit idle. The cache hits would be superb and the server would fall over.
Following load alone is the round-robin failure from Section 4 — perfectly balanced, and destroying reuse.
So a router scores rather than optimises. Something like:
score(server) = w₁ × prefix_match − w₂ × cache_utilisation − w₃ × queued_tokens
Send the request to the highest score. The weights encode the trade-off: a cache hit is worth a great deal, but not worth sending work to a server that is nearly out of memory.
The useful mental model is that locality is treated as a strong preference that yields under pressure. Prefer the warm server until it is loaded enough that the hit is no longer worth having — at which point the request goes elsewhere and, incidentally, warms a second replica.
One more tension worth naming, because it shapes where this field is going.
The two phases of a request have different needs. Prefill is a large burst of compute that wants a server with spare arithmetic capacity. Decode is a long, memory-bandwidth-bound trickle that wants a server with free cache memory and room in its batch. A single request needs both, in sequence — and routing it to one server means choosing a machine that must be good at both, at whichever moment it happens to arrive.
Worse, the two interfere. A large prefill landing on a server occupies the GPU long enough that every request already generating there sees its token stream stutter. So routing a long prompt to a replica is not just a load decision about that request; it degrades the experience of everyone currently mid-response on that machine.
Some systems respond by pushing this to its logical conclusion and running prefill and decode on separate pools of GPUs, routing each phase independently and transferring the cached state between them. That is a substantial architectural change and a topic in its own right — but the tension that motivates it is visible right here, in a router trying to make one placement decision for two incompatible workloads.
Everything above depends on estimating what a request will cost. The router can measure half of that.
Input length is known. The prompt is right there. Prefill cost and initial memory footprint can be estimated well.
Output length is not. Whether the model produces 20 tokens or 4,000 is not knowable until it happens. And output length determines how long the request holds memory and how many decode steps it consumes — which is most of its total cost.
So routing is a decision under uncertainty, and systems handle it with estimates rather than answers: a declared max_tokens gives an upper bound, historical averages per endpoint or per customer give a prior, and continuous feedback lets the router correct as servers report their real state.
The honest framing is that an LLM router does not compute optimal placement. It makes a good decision from partial information and relies on fresh feedback to correct.
Multi-turn chat deserves a specific mention, because it is where locality pays off most and is easiest to capture.
Each turn of a conversation re-sends everything before it. Turn five’s prompt is turns one through four plus a new message — meaning its prefix is exactly what the server computed on turn four. Route turn five back to that same replica and almost the entire prompt is a cache hit.
Route it elsewhere and the whole conversation is prefilled again from scratch, and it gets worse every turn, because the history keeps growing.
So routing by session identity is a large and cheap win. Two caveats worth knowing. It creates hot spots — a few very active conversations can concentrate load — so stickiness should still yield to a genuinely overloaded server. And it is only ever an optimisation: if the replica has restarted or evicted the entry, the request must still be correct, just slower.
The router is on the critical path of every request, so it has a budget: single-digit milliseconds. Decisions must come from state it already holds, not from queries made at request time.
That means it maintains a continuously updated view of the fleet, refreshed by servers reporting their own state — cache utilisation, queue depth, running sequences, readiness. Inference engines expose these as metrics, which is what makes this style of routing practical rather than theoretical.
Two properties follow, and both matter later. The router’s view is always slightly stale, so routing has to tolerate acting on information that is a moment out of date. And the router must know which servers exist right now, which is a harder requirement than it sounds.
TTFT at p95, not the mean. Routing failures show up in the tail. A router that is right 90% of the time and catastrophically wrong the rest looks fine on average.
Prefix cache hit rate across the fleet. If it is far below what a single server achieved, routing is destroying locality.
Load imbalance. Compare cache utilisation across replicas. A wide spread means the router is not seeing real load — which usually means it is counting something instead of measuring it.
Queue wait time, separated from prefill time. They have different causes and different fixes.
We now have a router that makes decisions from live server state: which replicas have memory, which are backed up, which hold useful cached prefixes.
All of which assumes something we have never examined. The router holds a list of inference servers, believes they are alive, and has current state for each one.
Where does that list come from?
In production it is not fixed, and it is not stable. Replicas are added when traffic rises and removed when it falls. Servers crash and are replaced. Deployments roll out new model versions, draining old replicas while new ones load weights — a process that takes minutes, during which those servers exist but must receive nothing. Hardware fails. Nodes are drained for maintenance.
The fleet the router is routing to is being continuously created, destroyed and reshaped underneath it. Something has to run those inference servers, place them on machines with the right GPUs, watch their health, replace the dead ones, add capacity under load, and publish an accurate list of which are ready to receive traffic — updated constantly and reliably enough that the router can trust it.
What actually operates a fleet of GPU servers, and how does it keep an accurate picture of what is running?
That is the orchestration layer, and it is the next piece of the stack.
1. What really happens when you click ‘Send’ on ChatGPT — A journey through modern AI Infrastructure
2. What Do You Do With a Model That’s Too Big for Your GPU? — Quantization, Sharding and Parallelism Explained
3. How Does One GPU Serve Hundreds of Users at the Same Time? — Inside an LLM inference server
4. The KV Cache Explained: Why Long Conversations Get Expensive — How LLMs remember context without recomputing everything
5. Why Is Your LLM Recomputing the Same Prompt 1,000 Times a Day? — Prefix caching, radix trees and block hashing explained
6. Why Traditional Load Balancing Breaks for LLMs — Building an LLM-aware router
7. Kubernetes for LLM Inference: How AI Workloads Run Across a GPU Cluster(Next Article)
8. LLM-D Explained — How modern AI infrastructure routes, schedules and scales LLM inference
9. Inside a Modern AI Inference Platform — The full stack end-to-end
Sources
Why Traditional Load Balancing Breaks for LLMs was originally published in Towards AI on Medium, where people are continuing the conversation by highlighting and responding to this story.