AI inference bills do not scale gracefully. Once your application shifts from single-turn Q&A to agentic sessions — where each user fires off dozens of long, context-heavy prompts in sequence — your GPU costs can jump 10 to 100 times overnight. The industry has been waiting for an open-source answer that does not involve rewriting your stack or signing an enterprise deal. On October 20, at the PyTorch Conference North America, that answer shipped: llm-d 1.0, a Kubernetes-native distributed inference platform backed by IBM, Red Hat, Google, NVIDIA, and 500-plus contributors across 25 organizations. IBM Research already benchmarked it at 11.5x lower inference cost versus an all-proprietary configuration. Here is what actually makes it work, and how to get it running.
Why Plain vLLM Falls Short at Scale #
vLLM is excellent. If you are running a single model on one GPU host for under 50 concurrent users, it is still the right tool. The problem surfaces when you scale out. With multiple vLLM replicas behind a load balancer, requests distribute randomly across pods. Each pod holds its own KV cache. A prompt that your system just computed three seconds ago on Pod A gets recomputed from scratch when it lands on Pod B. At 50 replicas and thousands of concurrent agentic sessions, you are burning GPU cycles recalculating context you already paid for.
llm-d solves this with KV cache-aware routing. Instead of randomly distributing requests, the llm-d Inference Scheduler hashes the prompt prefix and routes the request to whichever replica already has that prefix cached. Cache hits eliminate redundant computation. The savings compound fast at scale — which is where the 10x claim actually comes from. It is not magic; it is cache locality applied to LLM inference.
What 10x Actually Means #
Let us be specific about the cost claim, because vague promises get people burned. The 10x reduction applies to context-heavy agentic workloads with high prompt reuse — think coding agents, customer support bots with long conversation history, or AI systems that repeatedly reference the same documents. IBM Research hit 11.5x lower cost exploring quantum error correction hardware, where the prompts share substantial structure across requests.
For general-purpose chat with diverse prompts and low repetition, the gains are still real but more modest — roughly 3x output throughput in standardized benchmarks. The peak demonstration is striking: llm-d serving 6.6 million output tokens per minute with 3,000 concurrent coding agents and zero preemptions. If your workload looks anything like that, the math on switching is straightforward.
Deploy llm-d in 4 Steps #
Before you start, verify your prerequisites: Kubernetes 1.29 or newer (1.33+ recommended), cert-manager installed, Gateway API CRDs v1.3.0+, the NVIDIA GPU Operator, and the standard Helm/kubectl toolchain. You will also need a HuggingFace token for model access.
Step 1 — Install the NVIDIA GPU Operator (skip if already running in your cluster):
helm repo add nvidia https://helm.ngc.nvidia.com/nvidia
helm repo update
helm install gpu-operator nvidia/gpu-operator \
--namespace gpu-operator --create-namespace
Step 2 — Add the llm-d Helm repository and install:
helm repo add llm-d https://llm-d.github.io/llm-d-deployer
helm repo update
helm install my-llm-d llm-d/llm-d \
--namespace llm-d --create-namespace
The default chart deploys a sample Llama model. Disable that to bring your own:
helm upgrade -i my-llm-d llm-d/llm-d \
--set sampleApplication.enabled=false \
--namespace llm-d
Step 3 — Configure your values file. The installer accepts a custom values.yaml for model selection, replica counts, KV cache sizing, and namespace. The official quickstart includes example files for common topologies:
./llmd-installer.sh --values-file ./examples/base.yaml --namespace llm-d
Step 4 — Verify the endpoint. Once pods are running, the Kubernetes Inference Gateway exposes an OpenAI-compatible API endpoint. Point your existing application at it and you are done — no client changes required.
llm-d vs. Plain vLLM: The Decision #
This is not a competition. Use llm-d when your deployment already outgrows a single node. Stay with standalone vLLM when it has not.
Switch to llm-d if:
- 50 or more concurrent users
- Three or more vLLM replicas needed
- Agentic or context-heavy workloads (prompt reuse is high)
- Multi-tenant cluster with different priority tiers
- Target throughput exceeds 60,000 tokens per second cluster-wide
Stick with plain vLLM if:
- Single GPU node
- Under 50 concurrent users
- Offline batch processing or experimentation
- Prototype, internal tool, or benchmark setup
Production Architecture in One Paragraph #
The mental model is simple: vLLM is to llm-d what Linux is to Kubernetes. vLLM handles the inference; llm-d handles the orchestration. The Inference Scheduler routes based on KV cache locality, load, SLA tier, and prefill/decode phase. The DisaggregatedSet controller manages prefill and decode replicas as a single logical unit, so rolling updates move both phases together without dropping context. Multi-tenant QoS applies priority and fairness-aware flow control so latency-sensitive requests do not starve under burst load. The router stays highly available with hot standby. The whole thing validates across vLLM and SGLang, on-premise and cloud.
The Bottom Line #
If you are already running Kubernetes with a multi-GPU fleet and your inference costs are climbing, llm-d 1.0 is not a science experiment — it is production-ready infrastructure backed by some of the most credible names in open-source. The KV cache routing alone justifies the operational overhead once you cross the three-replica threshold. The zero-downtime upgrade path for agentic sessions solves one of the nastiest operational headaches in production AI. Start with the quickstart, benchmark against your actual workload, and check the official documentation for tuning guidance. The 10x number is real — for the right workload.