Today at the PyTorch Conference, we released llm-d 1.0, an open-source, Kubernetes-native stack for distributed LLM inference. Co-founded by Google with Red Hat, IBM and others, we worked with a community of more than 500 contributors over 25 organizations on this Cloud Native Computing Foundation (CNCF) Sandbox project, bringing together two important communities: PyTorch, where vLLM and SGLang serve models, and Kubernetes, where distributed systems run and scale in production.
llm-d 1.0 arrives at a time when agents are changing the nature of inference. A year ago, most inference traffic was chat: one prompt in, one response out. Today, workloads are shifting to multi-step agents and reinforcement learning (RL) rollouts that fan out across tools, share long prefixes, and arrive in bursts. Serving them on a single machine — or behind a generic round-robin load balancer — breaks down fast. Replicas recompute the same prompts, accelerator memory fills up with duplicate state, and one bursty agent loop can stall interactive users.
llm-d solves this for teams serving their own models by turning a cluster of model servers into a coordinated serving pool. During the PyTorch Conference keynote, we showed this in action with 1,000 sandboxed agents on Agent Substrate on GKE hitting a single llm-d endpoint across vLLM, SGLang, and a mixed fleet of Google Cloud TPUs and GPUs. As traffic scaled across the three stages below, cache-aware routing and tiered KV cache kept each agent's steps on the replicas holding its context, while flow control queued bursty agent loops so interactive requests stayed fast.
llm-d already delivered prefix- and load-aware routing, KV cache off, prefill/decode disaggregation, wide expert parallelism and batch inference across Google Cloud TPUs and GPUs. These capabilities provide:
The components are designed so that you can compose them into your own bespoke production environment. Keep your engine, model and cluster, and adopt only the serving components and configurations you need.
Across the areas below, Google engineers worked with the llm-d community to connect upstream Kubernetes primitives, PyTorch serving engines, and accelerator-aware scheduling into a composable stack.
One endpoint, many accelerators
An agent fleet is not one workload. Interactive turns need low latency, background loops and reinforcement learning (RL) rollouts need high throughput, and demand swings by the minute. llm-d puts your accelerator pools behind a single endpoint, with one router, one KV cache index, and one flow-control policy in front of the whole fleet. Agents call a single API; the router picks the replica for each step and keeps a session's context on the replicas that already hold it. Adding capacity, mixing accelerator generations, or moving a model to a different pool becomes a platform change, not an agent rewrite. TorchTPU brings Google Cloud TPUs into that fleet, running PyTorch models natively inside upstream vLLM and SGLang without model rewrites. For more, see the llm-d on Google Kubernetes Engine (GKE) TPU guide.
Routing that remembers context
The llm-d router builds on Gateway API Inference, the Kubernetes project for inference-optimized routing that we launched. On GKE, GKE Inference Gateway runs the llm-d router as a managed service, simplifying installation and management and extending routing across regions. The router tracks the replicas holding each cached prefix and weighs load and predicted latency when it picks one. When accelerator memory fills up, rather than dropping it, llm-d uses vLLM’s OffConnector, Mooncake, or SGLang’s HiCache to move KV cache to host memory and then to SSD or Google Cloud Managed Lustre. This setup helps prevent prefix-eviction storms under heavy, multi-turn, and long-context conversational loads across mixed TPU and GPU fleets.
On an agentic coding workload on Ironwood TPU (TPU 7x), prefix-aware routing plus host memory offload cut median time-to-first-token (TTFT) by more than 85% and raised total throughput 2.4x versus a plain Kubernetes Service, as of October 2026 (source).
Disaggregation that upgrades safely
Prefill is compute-heavy and decode is memory-bound, so running both on the same replicas makes each wait on the other. llm-d runs prefill, decode, and multimodal encode as separate pools, with wide expert parallelism spreading large mixture-of-experts models across nodes. In 1.0, DisaggregatedSet — built on LeaderWorkerSet, the Kubernetes API for multi-host inference that we started — manages these pools as one unit. Rolling updates move prefill and decode together on matching versions while each phase scales independently. For more, see the disaggregated serving guide.
Async inference that runs alongside real-time inference
For latency-tolerant traffic, llm-d 1.0 introduces llm-d-async-processor, which pulls requests from queues like Google Cloud Pub/Sub or Redis and routes them alongside real-time traffic. Using queue and pool-dispatch gates, the processor checks available capacity and enforces strict priority lanes and quotas so background jobs run in slack capacity without starving interactive users. Time-slicing to put idle RL capacity to work
During RL training, accelerators often sit idle 40% to 60% of the time during rollout generation or weight updates. llm-d now supports Google Cloud TPU and GPU time-slicing to interleave async inference into those quiet cycles, raising aggregate accelerator duty cycles from a ~40% baseline to >70% (source).
Benchmarks you can trust
Tuning a distributed inference stack across routers, KV cache tiers, and prefill/decode pools takes repeatable measurement under realistic multi-turn load. Prism is llm-d’s open-source analysis dashboard for exploring published benchmarks across llm-d’s well-lit paths, backed by standardized llm-d-benchmark recipes and inference-perf workloads. Inspect the exact configurations behind the numbers, or upload runs from your own cluster to compare throughput, latency, and cost curves side by side.
llm-d builds on Kubernetes components you may already run: Gateway API, LeaderWorkerSet, Dynamic Resource Allocation, Prometheus, and OpenTelemetry. Running llm-d on Google Cloud brings these pieces together with managed infrastructure:
This GKE llm-d stack carries production traffic, and it has been tested at fleet scale:
Organizations running llm-d on Google Cloud are putting these same capabilities to work in different ways, for instance prefix-aware routing to raise cache hit rates, open interfaces that fit an existing service mesh, and flow control to hold latency for priority tiers when traffic spikes.
“At Snap, we are integrating llm-d into our production AI infrastructure to facilitate high-performance inference at scale. By employing prefix-cache-aware routing, we have achieved prefix cache hit rates ranging up to 75-80%. We appreciate the open-source nature of llm-d, as it enables seamless integration with our Envoy-based Service Mesh." - Vinay Kola, Senior Manager, Software Engineering, Snap Inc.
"At ZenteiQ, we use GKE Inference Gateway powered by llm-d to accelerate synthetic data generation for our sovereign Scientific Foundation Models. By leveraging load-aware and prefix-cache-aware routing, we have achieved over 4× higher aggregate inference throughput, increasing daily generated token capacity from 1.47 billion to 5.97 billion tokens per 1,000 TPU v6e accelerators. The open-source foundation of llm-d, integrated with GKE, enables us to efficiently serve multiple model families at scale without developing and maintaining a custom inference scheduler.” - Prof. Sashikumaar Ganesan, Founder, Zenteiq.ai
“Serving frontier models across interactive applications and high-volume APIs means handling steep traffic spikes without letting bursty workloads degrade latency for priority customers. With llm-d's prefix-cache-aware routing, flow control and disaggregated serving support we can enforce strict priority bands across our commercial tiers, automatically holding or shedding lower-priority requests when pools saturate so paid traffic meets our latency SLOs. Mistral is a proud supporter of open-source software, and we will continue to contribute back to llm-d and DisaggregatedSet projects.” - Mathis Felardos, Lucas Fernandes-Martins and Nikhil Thomas, Mistral, Inference Engineers and llm-d contributors
Teams serving their own models have been asking for an open stack they can take apart and adapt to their own infrastructure, with the cache-aware routing, traffic control, and upgrade controls required in production. llm-d 1.0 gives them that and a community to build with.