The distributed inference framework makes serving large models on your own infrastructure reliable and affordable for the agentic era.
Today at the PyTorch Conference North America, llm-d project contributors are announcing llm-d 1.0, a major milestone for open-source AI infrastructure. A Kubernetes-native distributed inference serving network, llm-d makes it possible to deploy inference stacks for any model, on any accelerator, in any cloud.
The release signals that llm-d has moved beyond demonstrating individual performance techniques and is ready to support the realities of production: sustained efficiency under many workloads, all while balancing service priorities amid growing demand. And with inference costs increasing, llm-d is meant to help bring model deployment in-house and deliver returns on AI investment.
The work has been developed with a growing open-source community that includes hardware providers, model developers, cloud platforms, and enterprise users. As lead maintainers of llm-d, Red Hat, IBM, and Google have helped turn advanced inference techniques into a coordinated system that enterprises can operate at scale. The three co-founded llm-d last year with CoreWeave and NVIDIA, and industry leaders including AMD, Cisco, Hugging Face, Intel, Lambda, and Mistral AI have joined as contributors along the way.
The result is a vendor-neutral alternative for organizations that want frontier-class inference performance without depending entirely on one proprietary model provider, cloud, or hardware platform.
“llm-d is ready for day 2 operations,” said Carlos Costa, Distinguished Engineer at IBM Research, llm-d co-founder and lead maintainer. “llm-d has been delivering state-of-the-art inference performance for production workloads. What 1.0 adds is the maturity to run it like any other critical service: a stable API, multiple tenants with competing priorities sharing the same infrastructure, and upgrades that don't interrupt operations. That is what we built llm-d 1.0 to do.”
A full production stack #
Since llm-d launched, the project has focused on five important approaches to improving AI inference: prefix-cache-aware routing, which directs a request to infrastructure that may already hold its shared context; KV-cache off, which expands the useful life of cached information by moving it across GPU, CPU, and storage resources; batch inference optimization, which groups work so infrastructure can process it more efficiently; prefill/decode disaggregation, which separates the work of reading a prompt from the work of generating a response; and wide expert parallelism, which distributes large mixture-of-experts models across many accelerators.
These ideas are not new to version 1.0. What is new is the level of integration and operational maturity around them.
With version 1.0, llm-d has matured into a stack capable of supporting day 2 operations: everything that must happen after an AI service is deployed. That includes serving workloads with competing priorities, managing different performance requirements, maintaining efficiency, prioritizing important services, and keeping the system dependable over time. A benchmark can show that one feature improves throughput, but a production system must keep delivering that performance when traffic changes, hardware fails, software is updated, or different users compete for the same resources. llm-d 1.0 is about making the individual techniques work together under those less predictable conditions.
One endpoint, different kinds of work #
Expanding on the core technical innovations that have already made llm-d a powerful tool for distributed inference, llm-d 1.0 enhances support for multi-tenant quality of service, zero-downtime operations, and agent-aware serving, for sustained performance in common day 2 production operations. An enterprise AI service may receive requests from people and autonomous agents through the same endpoint, but those workloads behave differently. A person using a coding assistant expects a fast answer. An agent may send a large burst of requests, while using a tool, and then return for another round of model calls.
With flow control, llm-d 1.0 manages competing priorities whether workloads share the same hardware or span a mix of accelerators. Flow control, fairness, and prioritization help prevent bursty workloads from overwhelming latency-sensitive services. Under heavy demand, the system can make more deliberate decisions about which requests should receive resources first. Requests can also remain pinned to the appropriate infrastructure across multiple turns. That keeps their prefix caches warm, increasing the chance that previously processed context can be reused instead of calculated again.
This matters for conversational assistants, but it is especially valuable for agents. Agents frequently return to the same code, documents, instructions, and conversation history while carrying out a task. Prefix-cache-aware routing sends each turn toward a replica that already holds the relevant context, while KV-cache off helps preserve more of that reusable information when GPU memory becomes constrained. This is what production readiness looks like in practice: serving a model more effectively and more cost-effectively by understanding the work arriving at that model and managing infrastructure accordingly.
Improving AI ROI with better infrastructure #
This production focus leads directly to the economics of AI. As AI moves into everyday business operations, inference has become a recurring expense. Agentic systems can increase that expense because a single assignment may involve many rounds of reasoning, tool use, evaluation, and revision. If every step requires a model to reread the same material, you’re repeatedly paying for work the system has already done.
In a recent test of an agentic coding workload, llm-d’s prefix-aware router captured 97% of the available cache reuse. By avoiding repeated computation and making more efficient use of existing hardware, techniques like this can reduce the cost of serving AI at scale. In another recent demonstration, IBM and Red Hat ran 3,000 software-engineering agents concurrently on a self-hosted deployment for 5 to 10 times less than the actual price of serving the same open model as a service.
When an organization operates its own infrastructure, the infrastructure becomes part of a cost structure that can be managed. By combining open-weight models with an open serving stack, llm-d gives organizations another option for controlling the relationship between AI consumption, infrastructure spending, and business value.
Connecting the open-source AI Ecosystem #
llm-d works with inference engines such as vLLM or SGLang, adding the distributed routing and operational capabilities needed to turn model servers into production services.
That also gives llm-d an important role in the broader open-source ecosystem. PyTorch is closely associated with the people developing AI models and applications. The Cloud Native Computing Foundation (CNCF), which hosts llm-d as a sandbox project, focuses more heavily on deploying and operating systems in production. The llm-d project is a bridge between these worlds.
“llm-d unifies the PyTorch and CNCF ecosystems by bringing Kubernetes-native orchestration directly to vLLM and modern inference engines,” said Stephen Watt, vice president, office of the chief technology officer, Red Hat. “It delivers high-performance, distributed inference with the reliability and scale enterprises need — all within an operational model they already know.”
IBM’s contribution to llm-d reflects a long-standing approach: Take ideas that can improve computing and turn them into open technologies that organizations can use in the real world.
With llm-d 1.0, that work has helped produce an inference stack designed not only to perform well, but to keep performing when AI becomes critical infrastructure. Open-source AI inference is ready to be operated, governed, and optimized in production.
Related posts #
- Technical noteKaoutar El Maghraoui
- Preparing chip designers for the nanostack era ReleasePeter Hess
- Making AI-driven chip design affordable with open agent factories and distributed inference at scale NewsPeter Hess
- How distributed inference keeps large AI models moving ExplainerPeter Hess