llm-d 1.0: The Inference Layer vLLM Was Missing

Llm-d, the Kubernetes-native fleet coordination layer for LLM inference built jointly by Red Hat, IBM, Google, CoreWeave and NVIDIA, reached version 1.0 on October 20 at PyTorch Conference North America, adding multi-tenant quality of service, zero-downtime rolling upgrades and prefix-cache-aware routing. IBM benchmarks serving 3,000 concurrent coding agents on H100 hardware captured 97% of available cache reuse, delivering 6.6 million output tokens per minute at peak, while AWS reported a 70% throughput improvement over standard vLLM on SageMaker HyperPod. Self-hosting with llm-d on H200 hardware runs $0.05–$0.08 per million input tokens and $1.92–$3.55 per million output tokens versus $3–$15 per million tokens for managed inference APIs, and the CNCF-hosted open source project is already in production at DigitalOcean, AWS, Roblox and JetBrains.

If you’re running vLLM across multiple nodes and routing with a round-robin load balancer, you’re leaving cache hits — and money — on the table. On October 20, llm-d hit 1.0 at PyTorch Conference North America, the Kubernetes-native fleet coordination layer for LLM inference built jointly by Red Hat, IBM, Google, CoreWeave, and NVIDIA. It’s open source, CNCF-hosted, and already in production at DigitalOcean, AWS, Roblox, and JetBrains. Teams migrating from managed inference APIs are reporting 5–10x reductions in spend. Here’s what changed in 1.0, how the architecture works, and when it’s worth adopting. It Does Not Replace vLLM — It Sits Above It The first misconception to clear: llm-d is not a competing inference engine. vLLM handles the actual computation — running attention, managing GPU memory, generating tokens. llm-d handles what comes before that: routing requests intelligently, managing multi-tenant priorities, keeping cached context alive across agent turns, and deploying updates without downtime. Think of it as the Kubernetes to vLLM’s container runtime. Your existing vLLM or SGLang instances continue doing what they do; llm-d adds the coordination layer that turns a pile of individual servers into a production-grade serving fleet. Red Hat’s framing is precise: the choice isn’t between vLLM and llm-d — it’s about when you need both https://www.redhat.com/en/blog/demystifying-llm-d-and-vllm-race-production . What 1.0 Actually Ships Prior versions were useful for experimentation. v1.0 adds the operational maturity to run llm-d as critical infrastructure. Three capabilities were hardened: Multi-tenant quality of service. You can now run internal and external workloads on the same fleet without one set of requests starving the other. The new flow control layer supports priority- and fairness-aware scheduling with controlled load shedding — a probabilistic admitter handles burst traffic gracefully. Carlos Costa, IBM Distinguished Engineer and llm-d co-founder, stated: “llm-d is ready for day 2 operations — stable API, multiple tenants, and upgrades that don’t interrupt operations.” Zero-downtime operations. The router now has hot standby. The new DisaggregatedSet controller manages prefill and decode replicas as coordinated units, enabling rolling upgrades without maintenance windows. Model updates no longer require taking traffic offline. Agent-aware serving. For agentic workloads — coding assistants, RAG pipelines, multi-turn reasoning — context reuse is the biggest lever on cost. llm-d 1.0 routes follow-up requests back to the replica holding the relevant KV cache. In IBM benchmarks serving 3,000 concurrent coding agents on H100 hardware, prefix-cache-aware routing captured 97% of available cache reuse https://research.ibm.com/blog/llmd-1-0-is-ready-for-production-inference-serving , delivering 6.6 million output tokens per minute at peak. The Cost Case Managed inference APIs charge $3–$15 per million tokens depending on provider and model tier. Self-hosting with llm-d on H200 hardware runs $0.05–$0.08 per million input tokens and $1.92–$3.55 per million output tokens — a 5–10x reduction at scale. AWS’s numbers on SageMaker HyperPod with llm-d show a 70% throughput improvement over standard vLLM, combining prefix-cache routing with NIXL over Elastic Fabric Adapter. That said, “self-hosting” isn’t free. You’re trading API billing for GPU infrastructure, Kubernetes overhead, and engineering maintenance. The economics favor self-hosting somewhere north of a few million tokens per day, depending on your existing infrastructure. Who Is Already Running It DigitalOcean built its distributed inference service on llm-d specifically to serve the trillion-parameter Kimi K3 model across a heterogeneous mix of AMD and NVIDIA hardware — the multi-accelerator scenario a single vLLM instance can’t handle. AWS integrated llm-d into SageMaker HyperPod https://llm-d.ai/blog/llm-d-v1.0-state-of-the-art-inference-ready-for-day-2 for disaggregated inference. Roblox, JetBrains, Mistral, and Cohere are also among early adopters. When You Actually Need It llm-d makes sense when you’re running more than one inference replica and any of these apply: you have mixed workload priorities, you’re serving agentic applications where context reuse matters, you need model rollouts without offline downtime, or you’re mixing hardware types. If you’re on a single vLLM node, plain vLLM is still the right answer — the coordination overhead is not justified at that scale. If you’re paying $20K or more per month in managed API costs and have the infrastructure maturity to run Kubernetes workloads, the math warrants a serious evaluation. llm-d is a CNCF sandbox project — that’s real governance and solid momentum, but it hasn’t graduated yet. Teams with low tolerance for early-stage tooling should watch the next two quarters. For everyone else, the v1.0 docs are at llm-d.ai https://llm-d.ai/docs/architecture/latest-release and the release ships signed, verified tags on GitHub.