LLM-D Explained

Llm-d is a Kubernetes-native distributed inference serving stack, released open source under Apache 2.0, that adds inference-aware orchestration on top of vLLM and Kubernetes without replacing either. The stack introduces the InferencePool as a first-class Kubernetes object for grouping model-serving Pods and an endpoint picker that replaces round-robin load balancing with scheduling that scores prefix-cache locality, KV cache load and SLA, plus prefill/decode disaggregation across separate GPU pools. The project is built by contributors from the Kubernetes and vLLM projects and targets deployments large enough to need coordination across many servers.

TL; DR Kubernetes knows which Pods exist but nothing about what is inside them. A custom router knows what is inside but cannot create or scale them. llm-d closes that gap. It is a layer , not a replacement: vLLM stays the engine, Kubernetes stays the orchestrator, and llm-d makes the orchestration inference-aware. The InferencePool makes a group of model servers a first-class Kubernetes object, and an endpoint picker replaces round-robin with scheduling that scores prefix-cache locality, KV cache load and SLA. Prefill/decode disaggregation runs the two phases on separate GPU pools so each can be sized for the resource it actually consumes — at the cost of transferring KV cache between them. Cache stops being trapped in one Pod’s VRAM. It can be offloaded to host memory and disk, or shared across instances with a global index. This is scale machinery. Below a certain size it is complexity you do not need. A production LLM deployment ends up with two components that each hold half the truth. Kubernetes knows which inference servers exist, which are ready, and where they run. It knows nothing about what is happening inside them — not tokens, not batches, not cache occupancy. To the control plane, a server at 95% cache utilisation and one at 10% are two healthy Pods. A custom router knows exactly that missing half. It tracks queue depth, cache pressure and which replicas hold useful cached prefixes. It cannot create a server, replace a dead one, or add capacity. So the platform’s two most important decisions are made by components that cannot see each other’s information. Kubernetes terminates a Pod during a rollout without knowing it held the hottest cache in the fleet. Its built-in load balancing distributes by connection, which is precisely the behaviour that fails for inference — so you bypass it with a router bolted on beside the platform rather than built into it. llm-d is an attempt to remove those workarounds by making the orchestration layer itself understand inference. llm-d is a Kubernetes-native distributed inference serving stack, built as a layer on top of three things that already exist: vLLM as the inference engine, Kubernetes as the orchestrator, and the Gateway API Inference Extension as the routing foundation. It is open source under Apache 2.0, built by people from the Kubernetes and vLLM projects. The clarifications matter, because the name suggests more replacement than there is: It does not replace vLLM. Batching, the KV cache, PagedAttention, prefix caching — all of that continues to happen inside vLLM exactly as before. llm-d decides which vLLM instance handles a request and how those instances are deployed. It does not replace Kubernetes. Pods, Deployments, scheduling, probes and Services all still apply. llm-d adds inference-specific objects and behaviour on top. It does not make a single GPU faster. Everything it offers is about coordinating many servers. vLLM decides how a request uses a GPU. Kubernetes decides where servers run. llm-d decides which server, in what role, holding what state. Start with the most concrete gap. A Kubernetes Service groups Pods behind a stable address and distributes traffic among them, effectively round robin. It works because it assumes the Pods are interchangeable — which, once each one holds a different cache, they are not. The Gateway API Inference Extension adds an object built for this: the InferencePool . It represents a group of model-serving Pods as a first-class thing the platform understands, with an associated model and an associated way of choosing between its members. The important difference is not the grouping. It is that an InferencePool comes with a pluggable decision-making component instead of fixed round-robin behaviour: That component is called the endpoint picker , and it is where the routing logic from a hand-rolled router now lives — inside the platform, using standard Kubernetes objects, rather than beside it. llm-d’s inference scheduler is its implementation of the endpoint picker, tuned for vLLM. It works in two stages that are worth separating because they answer different questions. Filters eliminate candidates that cannot serve the request at all — wrong model, not ready, wrong role. This is correctness. Scorers rank the survivors. This is optimisation, and it is where the signals become concrete. The scheduler uses operational telemetry from vLLM to make decisions that are aware of prefix-cache state, KV cache load, request role, and service-level objectives. In other words, exactly the signals that make LLM routing work — where the cached prefix already lives, how close a server is to running out of cache memory, how backed up it is — but sourced from the engine through a defined interface rather than scraped by something you built yourself. Teams can add their own scorers where the defaults do not fit. A fair question at this point: how does the scheduler know what any given Pod holds in its cache? It does not inspect them per request — that would be far too slow for something on the critical path. Instead the workers continuously report their own state, and the scheduler maintains a running picture from that stream, supplemented by knowing which prefixes it has recently sent where. This means its view is always slightly out of date, which is why locality is treated as a score to weigh rather than a fact to obey. The practical difference is not that better decisions become possible. It is that they stop being bespoke. The routing intelligence becomes a component of the platform with a supported way to extend it. Now the part that changes deployment architecture rather than just routing. A request has two phases with opposite hardware appetites. Prefill processes the whole prompt at once — a large burst of parallel work that is limited by compute. Decode generates one token at a time, reading the model’s weights from memory on every step, limited by memory bandwidth. Run both on the same server, as a normal deployment does, and two problems follow. Neither phase gets hardware suited to it. The server must be adequate at both, so it is optimal for neither. They interfere. A large prefill occupies the GPU long enough that every request currently generating on that machine stutters. One user submitting a long document degrades inter-token latency for everyone else on that server. Disaggregation separates them: one pool of GPUs runs prefill workers, another runs decode workers, and a request visits both. Each pool can now be sized, tuned and scaled for the resource it actually consumes, and prefill work no longer interrupts anyone’s token stream. The reported gains are meaningful — throughput improvements on the order of tens of percent on identical hardware, and better inter-token latency because the interference is gone. One thing that surprises people: both pools still hold the full model. A prefill worker and a decode worker each run the complete set of weights, because each performs a full forward pass — one over many token positions, the other over a single position. Splitting the phases does not split the model. So disaggregation does not save weight memory, and a naive split can even use more GPUs than the aggregated deployment it replaced. The gain comes from specialisation and from removing interference, not from a smaller footprint. If prefill happens on one machine and decode on another, something must travel between them. Prefill’s entire output is the KV cache — the attention state for every prompt token. Decode cannot start without it. So disaggregation requires transferring that cache from the prefill worker’s GPU memory to the decode worker’s, and for a long prompt that is gigabytes. This is the whole engineering problem of disaggregation, and it is why the idea is recent despite being obvious. It only works with a transfer path fast enough that moving the cache costs less than the interference it avoids. llm-d uses high-performance transport for this — NVIDIA’s NIXL over fast interconnects such as InfiniBand and RDMA — rather than ordinary networking. The consequence is a real precondition, not a footnote: Disaggregation pays off when prompts are long enough that avoided interference outweighs the transfer, and when the interconnect is fast enough to make the transfer cheap. Neither is automatic. For short prompts on commodity networking, an aggregated deployment can easily be the better choice. This is scale machinery, and it assumes infrastructure to match. Putting it together: Step 3 is LLM-aware routing, now inside the platform. Steps 4 and 5 are the disaggregation. Step 7 is what keeps the scheduler’s picture current. This is the most consequential idea in llm-d, and it resolves a limitation that has been accumulating through every layer below. Until now, cached attention state has been trapped. It lives in one Pod’s GPU memory. It cannot be used by another replica. It is evicted when that GPU’s pool fills. And it is destroyed entirely when the Pod restarts — a replacement Pod is functionally identical and economically much worse until it warms up. llm-d uses vLLM’s KVConnector interface to make the cache a pluggable, layered resource rather than a fixed property of one GPU. Two schemes are described: Independent offloading. A Pod’s cache spills beyond its GPU into host memory and local disk. Entries evicted from VRAM are demoted rather than deleted, and can be pulled back faster than recomputing them. This is local, cheap to operate, and adds capacity without adding GPUs. Shared caching. Cache is transferred between instances and backed by shared storage with a global index , so a prefix computed by one worker can be found and reused by another. That second one is the significant shift. It means the cluster can know where cached state lives, and the scheduler can route to it — or fetch it — rather than recomputing it because it happened to land on the wrong machine. The trade-off is stated plainly in the design: shared caching offers higher performance at the cost of a considerably more complex system to operate. Independent offloading is the low-effort option; the shared, globally indexed cache is the high-ceiling one. Worth tracing the arc, because it is the through-line of this entire stack. Attention state began as a within-request optimisation, so a token would not be recomputed on every step. Prefix caching made it reusable across requests on one server. Cache-aware routing made its location a factor in where requests go. And here it becomes a cluster-level resource — tiered, transferable, and able to outlive the process that produced it. Splitting prefill and decode creates a scaling problem that ordinary autoscaling cannot express. The two pools consume different resources and are stressed by different traffic. A workload of long documents and short answers is prefill-heavy. Long conversational generation is decode-heavy. And the correct ratio between the pools depends on that mix, which changes through the day. Scaling on a single aggregate metric cannot handle this — adding replicas to the wrong pool spends money without relieving the bottleneck. What is needed is an autoscaler that measures each instance type’s capacity, understands that different request shapes impose different loads, observes the current traffic mix, and computes the right combination of prefill and decode capacity to meet the SLO. llm-d describes exactly this as variant autoscaling , and it is an area still under active development — worth knowing as the direction rather than treating as settled. Disaggregation adds moving parts, and it is worth knowing where the new failure modes are. A decode worker dies mid-generation. Its KV cache goes with it, and in-flight responses on that worker fail. Prefill work already done for those requests is lost too — the same class of failure as before, now with an extra hop involved. A prefill worker dies before transfer. The request has produced nothing usable and must be retried from the beginning. The transfer path degrades. Because cache movement sits on the critical path between phases, interconnect problems appear as latency rather than as errors — which makes them harder to attribute. A disaggregated deployment on a congested network can perform worse than the aggregated one it replaced. The cache index goes stale. Shared caching depends on knowing where entries live. If that view drifts, the scheduler routes for hits that are not there, and the cost is silent: correct answers, quietly recomputed. None of these is disqualifying, but together they make the point that this architecture buys performance with operational complexity. An honest answer, because the ecosystem is loud. Probably not, if you run one model on a handful of GPUs, your traffic is modest, prompts are short, or you have no fast interconnect. A single vLLM deployment behind a simple router will serve you well, and it is far easier to operate. Probably yes, if you run many replicas across many GPUs, your workload has large shared prefixes, prompt and output lengths vary widely, you have fast interconnect between nodes, and GPU cost is large enough that a double-digit efficiency gain is worth real operational effort. llm-d is not the only implementation of these ideas — NVIDIA’s Dynamo pursues a similar architecture from a different angle. What they share is the underlying claim, and that claim is the durable part: At scale, inference cannot be treated as a stateless workload behind a load balancer. Where state lives has to become a first-class input to routing, scheduling and scaling. The specific projects will change. That conclusion will not. Step back and look at what has been built. A model too large for one GPU is quantized or sharded, its parameters spread across devices that coordinate through fast interconnect. An inference engine loads it once and serves many users at a time, batching their work continuously so that one expensive read of the weights produces many tokens. It keeps attention state in paged blocks so memory is not wasted, and retains prefixes so repeated prompts are not recomputed. A scheduler routes each request to the server most likely to serve it well, weighing cached state against memory pressure. Kubernetes runs the fleet — placing servers on the right hardware, replacing failures, gating traffic on readiness. And an inference-aware layer ties routing, scheduling and cache into one system that understands the workload, splitting requests by phase and letting cached state move between machines. Every architectural question this series raised now has an answer. Which leaves the question that only appears once the architecture is done. Building this system and operating it are different problems. The remaining questions are not about what to build: How do you size it before traffic exists? What do you put on a dashboard, given that GPU utilisation is misleading and averages hide the failures that matter? What does it cost per million tokens, and which decisions move that number most? When it degrades at three in the morning, how do you find out which of these layers is responsible? How do you run this thing? That is the last piece. 1. What really happens when you click ‘Send’ on ChatGPT — A journey through modern AI Infrastructure 2. What Do You Do With a Model That’s Too Big for Your GPU? — Quantization, Sharding and Parallelism Explained 3. How Does One GPU Serve Hundreds of Users at the Same Time? — Inside an LLM inference server 4. The KV Cache Explained: Why Long Conversations Get Expensive — How LLMs remember context without recomputing everything 5. Why Is Your LLM Recomputing the Same Prompt 1,000 Times a Day? — Prefix caching, radix trees and block hashing explained 6. Why Traditional Load Balancing Breaks for LLMs — Building an LLM-aware router 7. Kubernetes for LLM Inference: How AI Workloads Run Across a GPU Cluster 8. LLM-D Explained — How modern AI infrastructure routes, schedules and scales LLM inference 9. Inside a Modern AI Inference Platform — The full stack end-to-end Next Article Sources LLM-D Explained https://pub.towardsai.net/llm-d-explained-69f7d0b2b035 was originally published in Towards AI https://pub.towardsai.net on Medium, where people are continuing the conversation by highlighting and responding to this story.