{"slug": "how-llm-d-makes-the-most-of-the-hardware-you-already-have", "title": "How llm-d makes the most of the hardware you already have", "summary": "IBM Research and Red Hat used the open-source llm-d framework to deploy GLM-5.2, an approximately 753-billion-parameter mixture-of-experts model, on 544 NVIDIA H100 GPUs, serving up to 3,000 concurrent coding agents with zero preemptions and delivering over 6.6 million output tokens per minute at peak. The deployment costs 5 to 10 times less per token than equivalent commercial API pricing, according to the team, which aims to show that self-hosted open-weight models can handle agentic workloads on common hardware.", "body_md": "# How llm-d makes the most of the hardware you already have\n\nIBM Research and Red Hat used llm-d to deploy a 753B open model on H100 GPUs, serving thousands of concurrent coding agents at 5-10x lower cost than commercial APIs.\n\nAs AI agents become more capable, they are placing new demands on inference infrastructure. Unlike traditional chatbots, agentic systems like coding assistants repeatedly process massive contexts, reuse information across many interactions, and recruit parallel sub-agents that generate unpredictable bursts of activity. These workloads are dominated by reading and managing context rather than generating text, creating challenges for latency, memory, throughput, and cost.\n\nWith the [llm-d project](https://research.ibm.com/blog/donating-llm-d-to-the-cloud-native-computing-foundation), a collaborative effort led by IBM Research, Red Hat, and Google, and joined by other industry leaders, an open-source framework has been introduced to help serve LLMs at scale, addressing the challenges introduced by these new types of AI workloads. llm-d is filling a need that has arisen in the enterprise AI landscape: Amid rising per-token costs and the need to secure proprietary data, enterprises are increasingly looking for ways to deploy open models on their own infrastructure. Now, the llm-d framework is proving itself capable of processing today’s agentic traffic.\n\nIn a recent [demonstration](https://llm-d.ai/blog/serving-glm-5-2-agentic-workloads-on-llm-d) using benchmark workloads, the llm-d project showed that its open-source inference platform could efficiently serve large mixture-of-experts models on H200 GPUs for agentic workloads. Many organizations face the same challenge: scaling agentic workloads on their existing fleet. The goal was to show that this was possible on H100 GPUs, an accelerator commonly deployed in enterprise and cloud GPU fleets.\n\nUsing llm-d, the team deployed GLM-5.2, an approximately 753-billion-parameter open-weight mixture-of-experts model (~39B active), on 544 NVIDIA H100 GPUs. This model was chosen as an example of what platform operators may be deploying today. On a workload with hundreds of concurrent agentic sessions run with high context reuse, representative of production traffic patterns, the deployment delivered over 6.6 million output tokens per minute at peak on agentic benchmarks, serving up to 3,000 concurrent coding agents with zero preemptions.\n\nAt current cloud rental rates, self-hosting GLM-5.2 on H100s with llm-d costs 5 to 10 times less per token than equivalent commercial API pricing, with the largest savings on the input-heavy traffic patterns typical of agentic workloads.\"We set out to show that a self-hosted, open-weight model can run real agentic workloads at competitive interactive throughput on long-context agentic workloads, using the GPUs most organizations already operate,\" said Carlos Costa, Distinguished Engineer at IBM Research and llm-d maintainer. \"llm-d has already proven it can handle this at scale. We wanted to close the gap for a model of this size on a common, previous-generation fleet, and that's what we demonstrated here.\"\n\n## Understanding agentic traffic\n\nIn the earlier study, data from 219 real-world Claude Code sessions showed that serving agentic workloads requires a different approach than traditional chatbot inference. Coding agents spend most of their time reading and processing massive contexts over and over, and relatively little time generating long responses. These contexts often contain entire software repositories — a repetitive, heavy lift. The median request in the study carried roughly 195,000 tokens but produced only 317, meaning the computational challenge lies overwhelmingly in processing incoming context rather than generating text.\n\nBased on their demonstration, the llm-d team concluded that three characteristics define agentic workloads: extremely long contexts, heavy reuse of information from earlier turns, and bursts of parallel activity from sub-agents. The analysis found that the same information is processed again and again: 96% of main-agent requests reused at least 90% of a previous request's input verbatim. Efficient systems, by contrast, should cache and reuse previous computations rather than recomputing them from scratch. The study also showed that more than half of all requests arrived in groups of concurrent sub-agent tasks without advance notice, creating sudden surges in workload that serving systems must handle without sacrificing responsiveness.\n\n## How llm-d can help\n\nllm-d combined six capabilities to reduce repeated context processing, preserve cache reuse under load, and scale prefill and decode capacity independently.\n\nBecause so much of the computation in agentic serving goes to reprocessing context the system has already seen, prefix-aware routing directs each request to the server that already holds its cached context, so the system reuses prior computation instead of starting from scratch. On the CyberGym agentic benchmark, switching from optimized approximate routing to precise prefix matching delivered 79% higher throughput and 67% lower time-to-first-token. Tiered key value (KV)-cache management extends the working cache into CPU DRAM so that useful prefixes survive GPU memory pressure rather than being evicted and recomputed. When the best cache match is on another server, peer-to-peer (P2P) KV-cache sharing pulls it from that peer instead of recomputing the prefix locally. Together, these capabilities reduce time-to-first-token and preserve GPU capacity by avoiding repeated prefill work, even when requests move between servers.\n\nA 753-billion-parameter model does not fit on a single server. llm-d handles this with wide expert parallelism with data-parallel attention, which distributes the model across nodes while avoiding the KV-cache replication that tensor parallelism would require for GLM-5.2's multi-head latent attention. Prefill/decode disaggregation then separates context processing from token generation into independent pools, each tuned for its workload, that scale based on demand. The two pools communicate through NVIDIA's NIXL zero-copy transfer library with zero observed transfer failures across the full benchmark program. This lets operators fit the model across existing H100 nodes and scale context processing separately from token generation as workload demand changes.\n\nFinally, multi-token prediction (MTP) generates multiple output tokens per forward pass, significantly increasing output throughput at high concurrency. Because MTP works on top of the other optimizations, the gains compound.\n\n\"llm-d is not just about a bunch of features,\" said Costa. \"It is about all of them working together, in production, on infrastructure that any organization can deploy. This is how it looks when it all comes together.\"\n\n## Ready to serve at scale\n\nThe team deployed GLM-5.2 on 544 H100 GPUs in a disaggregated topology with separate prefill and decode groups. The deployment supports an internal workload serving critical business needs, with hundreds to thousands of concurrent agents processing long-context, multi-turn tasks with high context reuse. The system is built to handle sustained production traffic at scale.\n\nTo characterize the system's limits and validate each capability under controlled conditions, the team ran structured benchmarks alongside the internal workload. Each benchmark used fresh prefixes and seeds to prevent prior cache states from influencing the results.\n\nAutomationBench tested raw concurrency by scaling from 2,000 to 3,000 concurrent coding agents. At 2,500 agents, the comfortable operating point, the deployment sustained 7,612 requests per minute with peak input throughput of 134.89 million tokens per minute and peak output of 6.05 million tokens per minute, all with zero preemptions. At 3,000 agents, output reached 6.60 million tokens per minute before approaching the system's serving boundary, with no preemptions or failures.\n\nSeparately, CyberGym evaluated long-context agentic completions: 400 concurrent agents, each processing 376K-character contexts over 10 turns. With the full routing stack, all 400 agent trajectories completed in 248 seconds at 967.7 requests per minute, with a p90 time-to-first-token of 17.59 seconds and p90 queue time of just 1.11 seconds. Local prefix hit rate reached 73.18%, up from 44.46% with approximate routing.\n\nAgentX tested interactive throughput with 128 concurrent agents processing ~195K-token contexts over approximately 15 minutes. The benchmark completed 7,251 requests at 7.7 requests per second with a p90 time-to-first-token of 5.77 seconds. During the run, 168 requests triggered a P2P lookup on another worker for reusable KV-cache blocks.\n\nUnderneath these results, the disaggregated data plane handled the full workload. NIXL completed 6.2 million KV-cache transfers, averaging 2.71 GiB per transfer and sustaining approximately 580 Gb/s at fleet p90, with zero transfer failures. The ratio of nearly one transfer per successful request is consistent with essentially all traffic traversing the split prefill/decode pipeline. The CPU-tier cache absorbed 2.53 PiB of prompt block stores and restored 2.16 PiB back to GPU, averaging 79.6 ms per restore. All benchmarks completed with zero serving errors.\n\nIn the measured cache decomposition, the stack served 85.2% of input tokens from cache and reduced uncached prefill to 14.8%, leaving more GPU capacity for new context and generation.\n\n“We were not chasing one peak throughput number. We wanted to understand whether an open model of this size could sustain the long contexts, reuse, and bursts that define agentic systems. Serving thousands of concurrent agents without preemption gives operators a credible path from a benchmark result to a production deployment.” said Maroon Ayoub, Senior Principal MLE, Inference Engineering, member of the Red Hat team.\n\n## What this means for platform operators\n\nTechnologies like llm-d put frontier-class open model serving within reach of organizations that already operate GPU infrastructure. Serving a model of this size at competitive throughput has often required newer-generation hardware or a fully managed API endpoint. This demonstration shows that with the right software stack, widely deployed H100 infrastructure can deliver strong results, giving operators of large H100 fleets a path to self-hosted inference on long-context agentic workloads without waiting for the next generation of accelerators.\n\n\"Operators of large H100 fleets can look at these numbers and see a viable path to serving frontier-scale models on their own infrastructure,\" said Costa. \"Agentic inference is a systems problem. The wins come from avoiding redundant work, routing to the right cache, scaling prefill and decode independently; not from raw GPU throughput only. \"\n\n“These results are not the ceiling for what H100 infrastructure can deliver. They give us a strong baseline, and every improvement to the open serving stack expands what the same fleet can support. That is the opportunity with llm-d: Software improvements compound the value of the hardware operators already have,” said Nili Guy, Senior Technical Staff Member, member of IBM Research team.\n\nThe team will continue to evolve llm-d in the open, applying lessons from internal deployments at scale to address real pain points across the community. Understanding how these systems behave under sustained agentic traffic shapes what the project builds next.\n\nllm-d is open source and a Cloud Native Computing Foundation sandbox project, with contributions from IBM Research, Red Hat, Google, and a growing community of contributors and adopters. The project provides deployment guides, validated configurations, and multi-platform support. Get started at [llm-d.ai](https://llm-d.ai/).\n\n## Related posts\n\n- ReleaseMike Murphy and Kim Martineau\n- [Introducing DocLang, a markup language for AI](/blog/doclang-ai-native-doc-standard) NewsKim Martineau\n- [From vision to reality: a unified neural solver for the power grid](/blog/gridfm-neural-solver-power-grid) ReleasePeter Hess\n- [Bringing a common language to AI evaluation](/blog/every-evaluation-ever) NewsKim Martineau", "url": "https://wpnews.pro/news/how-llm-d-makes-the-most-of-the-hardware-you-already-have", "canonical_source": "https://research.ibm.com/blog/running-open-models-on-h100-gpus-with-llmd?utm_medium=rss&utm_source=rss", "published_at": "2026-09-08 12:00:00+00:00", "updated_at": "2026-09-08 12:56:59.790796+00:00", "lang": "en", "topics": ["artificial-intelligence", "large-language-models", "ai-infrastructure", "ai-research", "ai-tools"], "entities": ["IBM Research", "Red Hat", "Google", "llm-d", "GLM-5.2", "NVIDIA H100", "Carlos Costa"], "alternates": {"html": "https://wpnews.pro/news/how-llm-d-makes-the-most-of-the-hardware-you-already-have", "markdown": "https://wpnews.pro/news/how-llm-d-makes-the-most-of-the-hardware-you-already-have.md", "text": "https://wpnews.pro/news/how-llm-d-makes-the-most-of-the-hardware-you-already-have.txt", "jsonld": "https://wpnews.pro/news/how-llm-d-makes-the-most-of-the-hardware-you-already-have.jsonld"}}