How distributed inference keeps large AI models moving

Distributed inference serving frameworks coordinate clusters of model servers above engines like vLLM and SGLang, routing requests based on queue depth, cache availability, and server roles. The article details six technical innovations — prefix-cache-aware routing, KV-cache offloading, flow control, batch inference optimization, prefill/decode disaggregation, and wide expert parallelism — and shows how they combine in llm-d, an open-source, Kubernetes-native inference framework. Prefix-cache-aware routing reuses KV cache for shared prompt prefixes, such as a chatbot's repeated system instructions or a coding assistant's recurring repository context, instead of sending requests to whichever server appears least busy.

How distributed inference keeps large AI models moving Distributed inference serving is an air traffic control tower for AI traffic, and agentic workflows make it more necessary than ever. When an AI model answers a question or completes a task, it performs inference, i.e., using a trained model to generate an output in response to some input. Doing that at scale, however, is more challenging than it sounds. Running a large language model on a GPU is one thing. Serving that model to thousands or even millions of people and AI agents at the same time is quite another. Requests arrive with different priorities and lengths, which have unique computing needs. Some users expect an immediate response to simple queries, while others submit large jobs that can run in the background. AI agent traffic adds a new pattern: Agents may send bursts of related requests, pause to call a tool, and then return with nearly the same context. Meanwhile, the system must make efficient use of expensive accelerators without creating long queues or repeatedly performing the same computations. This challenge resembles air traffic control. A busy airport cannot simply send each arriving aircraft to the first available runway. Controllers in the tower must account for the aircraft’s location, destination, size, and fuel needs, as well as the traffic already moving through the system. The challenge might be simple if all air traffic controllers had to do was keep every runway busy, but an airport isn’t a parking lot, and not every airplane can be treated the same way. Distributed inference plays a similar coordinating role for LLM inference, working a layer above model servers like vLLM https://github.com/vllm-project/vllm and SGLang https://github.com/sgl-project/sglang , where it orchestrates and optimizes real-world traffic. While those serving engines run models efficiently on individual servers, a distributed inference serving framework coordinates groups of model servers across a cluster. Its router can consider queue depth, cache availability, and server roles before deciding where a request should go. To understand how distributed inference works, we’ll look at six notable technical innovations—prefix-cache-aware routing, KV-cache offloading, flow control, batch inference optimization, prefill/decode disaggregation, and wide expert parallelism. We’ll also see how each comes together in llm-d, an open-source, Kubernetes-native inference framework. Prefix-cache-aware routing: Sending work where context already lives When a language model reads a prompt, it produces an internal representation of the words and their relationships, storing much of this intermediate state in a key-value cache, or KV cache. If a later request begins with the same text, or even portions of the same text, the model may be able to reuse that stored computation rather than re-process the shared text. For example, if an existing prompt is “What is the capital of Kansas?” then later on you ask, “What is the capital of California?” the KV cache for the "what is the capital of" portion is reused. This opportunity arises frequently in real workloads. A chatbot receives the same system instructions at the beginning of every conversation. A coding assistant repeatedly examines the same repository. An AI agent may revisit a long record of previous steps while testing different solutions. In each case, much of the prompt remains unchanged from turn to turn. A conventional load balancer may send the next request to whichever server appears least busy, whether or not its cache contains the relevant context. As a result, the model must reread material that another server has already processed. Prefix-cache-aware routing https://llm-d.ai/docs/architecture/advanced/kv-management/prefix-cache-aware-routing instead asks two questions: Which server already holds the greatest portion of this prompt’s prefix, and which server has enough capacity to handle the request? The llm-d router uses cache information and operating conditions to score the available model servers and select a suitable destination. There is both an approximate implementation that includes a lightweight index of recent routing decisions, and a precise implementation based on actual token data and cache-state updates from model servers. If an aircraft already has its passengers, baggage, and ground crew waiting at one gate, sending the plane to another gate would introduce needless work. In the same way, a distributed inference system tries to send a request to the model server that already contains the required context. The payoff is less redundant prefill computation, lower latency, and greater throughput. In a recent demonstration from the team behind llm-d, prefix-aware routing captured 97% of the available reuse https://llm-d.ai/blog/serving-glm-5-2-agentic-workloads-on-llm-d in the tested workload. More broadly, the technique is especially useful for agentic systems because their requests often share long instructions, source material, conversation histories, and tool outputs. llm-d was launched in May 2025 as a collaborative effort by founding members IBM Research, Red Hat, Google Cloud, NVIDIA, and CoreWeave. Several founding partners also took part from the start, including industry leaders like AMD, Cisco, Hugging Face, Intel, Lambda, and Mistral AI, and academic supporters at the University of California, Berkeley, and the University of Chicago. IBM Research, Red Hat, and Google Cloud continue to lead llm-d's development today, and its community of partners and contributors has grown considerably. In March 2026, the founders donated llm-d https://research.ibm.com/blog/donating-llm-d-to-the-cloud-native-computing-foundation to the Cloud Native Computing Foundation as a sandbox project, to further the mission of deploying inference stacks for any model, on any accelerator, in any cloud. KV-cache offloading: Extending a model’s working memory Cache-aware routing is most valuable when cached information remains available. The problem is that although accelerator memory is fast, its space can be quickly filled by long prompts, many concurrent users, and extended agent sessions. When space runs short, a model server may evict older KV-cache entries. If a later request needs that material, the model must calculate it again, wasting accelerator cycles rebuilding context that existed moments earlier. Distributed inference addresses this problem through KV-cache offloading https://llm-d.ai/docs/architecture/advanced/kv-management/kv-offloader . Rather than treating accelerator memory as the only place where cached context can live, it chooses an offloading target based on workload requirements. The most immediately useful entries can remain in high-bandwidth accelerator memory, while other entries can move to larger but slower tiers such as CPU memory, local solid-state storage, or a shared file system. A cache index tracks what is stored and where, allowing the routing layer to incorporate offloaded data into its decisions. Retrieving cached information from CPU memory or storage is slower than reading it directly from accelerator memory, but it is much faster and less expensive than recomputing the entire prompt. Offloading therefore expands the system’s effective cache capacity beyond the physical memory available on its accelerators. This matters most for workloads with long, reusable contexts and many simultaneous sessions. In the same llm-d demonstration mentioned in the previous section, GPU memory supplied about 90% to 92% of reusable prefill prefixes for 128 concurrent requests, with CPU offloading helping to preserve prefixes beyond GPU capacity. In another demonstration https://research.ibm.com/blog/running-open-models-on-h100-gpus-with-llmd , CPU offloading helped the system sustain high throughput and low time to first token across a long, agent-driven benchmark. Because offloading runs during prefill and writes to slower tiers happen asynchronously, these gains come without slowing token generation. Flow control: Deciding what moves and when Routing answers where a request should go. But under load, a system also has to decide which requests move first — and which ones wait. When a burst of traffic arrives at once, treating every request identically means a latency-sensitive query can end up stuck behind a backlog of lower-priority work. Air traffic controllers face the same problem and solve it with traffic flow management. When demand exceeds capacity, they meter departures, sequence arrivals, and hold lower-priority traffic on the ground so that a full airliner on final approach isn't delayed behind a cargo run that could just as easily wait a few minutes. Distributed inference needs the same discipline. llm-d's flow control governs how requests are admitted and ordered as they enter the system, enforcing fairness across users and priority across workloads. Latency-sensitive interactive traffic can be protected, while bursty or deferrable work is paced to fit the capacity that remains. This keeps a single noisy tenant or a sudden surge of agent activity from degrading response times for everyone else. This becomes especially important with agentic traffic, which is uneven by nature: An agent may sit idle, then fire a burst of related requests, then pause to call a tool. Flow control absorbs that irregularity, smoothing bursts into an orderly stream rather than letting them overwhelm the cluster. Batch inference optimization: Filling spare capacity without clogging the system Not every inference request needs an immediate answer. Research evaluations, document processing, synthetic-data generation, and large-scale testing may involve thousands or millions of prompts that can be completed over time. These batch jobs can improve hardware utilization by using capacity that might otherwise sit idle, like routing air freight to unused runways. Handled poorly, batch jobs can flood the system and delay interactive traffic. A batch workload should not make a person wait unnecessarily for a coding assistant or customer-service response. llm-d supports batch and offline inference https://llm-d.ai/docs/architecture/advanced/batch through two complementary components. A Batch Gateway provides an OpenAI-compatible interface for submitting, tracking, and managing batch inference jobs. An asynchronous processor can pull individual requests from message queues, including Redis and Google Pub/Sub, and dispatch them to the llm-d router. This design separates job management from request placement. The batch layer determines how work enters the system, while the router still decides where each individual request should run. Batch jobs rely on flow control to pace their entry according to current load rather than releasing an entire job into the cluster at once. The result is faster batch processing, as well as a more orderly use of shared infrastructure. Organizations can keep accelerators productive while protecting the responsiveness of higher-priority applications. Prefill/decode disaggregation: Separating two different types of work Generating an LLM response involves two computational phases. During prefill, a model processes the input prompt and builds the KV cache. This phase can involve large amounts of parallel computation, particularly when a prompt contains tens of thousands of tokens. During decode, the model generates the response one token at a time. Decode is sequential and highly sensitive to memory access and communication delays, placing different demands on the hardware than prefill. A conventional serving system may run both phases on the same group of accelerators, which simplifies deployment but creates competition between dissimilar workloads. A large prefill job can interrupt or delay steady token generation, while hardware configured for responsive decoding may be a poor fit for processing very long prompts. Prefill/decode disaggregation https://llm-d.ai/docs/dev/architecture/advanced/disaggregation allows a distributed inference system to assign the phases to specialized workers and scale them independently. The router selects a prefill endpoint and a decode endpoint, then coordinates the transfer of the resulting KV-cache state between them. This separation makes the infrastructure more adaptable. A workload dominated by long documents may require more prefill capacity. A service supporting many active conversations may need more decode capacity. Instead of expanding the entire serving cluster, operators can add resources where the pressure actually exists. Disaggregation also reduces interference between the phases. Prompt processing can be optimized for throughput, while generation can be optimized for the consistent delivery of tokens. Prefill/decode disaggregation allows the two phases to scale independently, improving efficiency for variable workloads. Wide expert parallelism: Spreading a giant model across the cluster Some of today’s largest language models use a mixture-of-experts MoE architecture. Instead of applying every part of the model to every token, an MoE model contains many specialized neural networks, called experts. A routing mechanism selects only a small subset of experts for each token. This approach reduces the amount of computation required for each request, but it does not eliminate the need to store the model’s full set of weights. Large MoE models can require hundreds of gigabytes of memory just to hold all their weights, leaving less accelerator memory available for the KV cache and other parts of inference. Traditional tensor parallelism addresses the memory problem by dividing individual model operations across several GPUs, which can require frequent communication among the GPUs. For models using certain attention mechanisms, it may also create multiple copies of the same KV-cache data, reducing the capacity available for long contexts and concurrent users. Wide expert parallelism https://llm-d.ai/docs/dev/well-lit-paths/foundations/wide-expert-parallelism takes a different approach. It distributes the model’s experts across GPUs and, when necessary, across physical servers. When the model determines which experts should process a token, the system sends that token only to the GPUs holding the selected experts. Those experts perform their calculations, and the results return to the original processing path. Because this communication is sparse, the GPUs exchange the information needed for the selected experts rather than continuously synchronizing every model operation. In air traffic terms, the system does not require every specialist at the airport to work on every aircraft. It directs each task to the small number of specialists it actually needs. Memory efficiency is a significant advantage of this technique. Spreading expert weights across a wider group of accelerators leaves more memory on each GPU for the KV cache. The effect can be substantial. In a demonstration https://llm-d.ai/blog/serving-glm-5-2-agentic-workloads-on-llm-d this year from IBM and Red Hat, the wide layout stored about 65.1 GiB of expert weights per GPU, compared with 89.5 GiB in the tensor-parallel layout examined. On the same hardware, the wide configurations provided between three and nine times as much KV-cache capacity, depending on the topology. In this way, wide expert parallelism helps distributed inference systems like llm-d serve extremely large MoE models while preserving room for long prompts and many active sessions. It also complements prefill/decode disaggregation: One divides inference by phase, while the other distributes the model’s experts across the cluster. Used together, the two allow llm-d to match the organization of the hardware more closely to the structure of the model and the demands of the workload. Coordinating the whole inference system As with an airport’s system of air traffic controllers, ground crews, and flight crews, each element is valuable on its own but can’t run the operation on its own. Prefix-cache-aware routing identifies where reusable context resides. KV-cache offloading preserves more of that context across a hierarchy of memory and storage. Flow control manages incoming requests to maintain capacity for simultaneous users. Batch inference controls how large volumes of background work enter the system. Prefill/decode disaggregation sends fundamentally different phases of inference to resources suited to each one. And wide expert parallelism physically distributes large models across clusters to make the best use of accelerator memory. Together, they turn LLM serving from a collection of isolated model servers into a coordinated and distributed system. That distinction becomes increasingly important as AI moves beyond simple chat and toward agents that maintain long contexts, call tools, collaborate with other agents, and generate irregular bursts of traffic. The air traffic control analogy only goes so far, but its central lesson holds: Performance depends on coordination, not just raw capacity. Building more runways does not automatically make an airport efficient, and adding more accelerators does not automatically make inference fast or affordable. The system also needs to know where work should go, what can be reused, which tasks can wait, and which resources are best suited to each stage. That is the role distributed inference is designed to fill. Related posts - ExplainerPeter Hess - Demonstrating a theoretical separation between quantum computers and large language models https://research.ibm.com/blog/quantum-circuits-vs-llms ResearchSrinivasan Arunachalam, Arkopal Dutt, Hari Krovi, Rik Sengupta, and Ryan Mandelbaum - A rough guide for going back to the Moon https://research.ibm.com/blog/nasa-ibm-lunar-foundation-model ReleaseKim Martineau - Accelerated Discovery https://research.ibm.com/topics/accelerated-discovery - AI https://research.ibm.com/artificial-intelligence - Algorithms - How llm-d makes the most of the hardware you already have https://research.ibm.com/blog/running-open-models-on-h100-gpus-with-llmd NewsPeter Hess