{"slug": "introducing-amd-rocm-infera-scaling-goodput-for-agentic-ai-with-distributed", "title": "Introducing AMD ROCm Infera: Scaling Goodput for Agentic AI with Distributed Inference Orchestration", "summary": "AMD introduced ROCm Infera, an open-source distributed inference reference solution for large-scale deployments, claiming it can improve goodput per GPU for agentic AI workloads by up to 2.6× in internal testing. Infera coordinates engine instances across KV-aware routing, prefill-decode disaggregation, and KV-cache tiering to address orchestration challenges in multi-turn agentic systems. The code is available on GitHub, and Infera integrates with vLLM, SGLang, and ATOM on AMD Instinct GPUs.", "body_md": "# Introducing AMD ROCm Infera: Scaling Goodput for Agentic AI with Distributed Inference Orchestration[#](#introducing-amd-rocm-infera-scaling-goodput-for-agentic-ai-with-distributed-inference-orchestration)\n\nToday we are introducing AMD ROCm™ Infera, a distributed inference reference solution for large-scale deployments. Infera is a conductor for your inference GPU orchestra. Initial internal testing shows that Infera can improve goodput per GPU for realistic agentic workloads by up to 2.6×, as [detailed below](#preliminary-performance). AMD-native Infera is open source from day one, and the code is available at [github.com/AMD-AGI/Infera](https://github.com/AMD-AGI/Infera).\n\nIn this blog, you will learn why scaling beyond a single engine instance creates new orchestration challenges, how Infera addresses them, how it integrates with vLLM, SGLang, and ATOM, and what our preliminary performance results and roadmap show for agentic AI on AMD Instinct™ GPUs.\n\nModern inference engines optimize execution within an engine instance, but production systems rely on many instances for scale, availability, and workload specialization. As deployments grow, teams typically use conventional load balancing or custom logic to distribute requests.\n\nThis creates an orchestration gap, especially for agentic workloads. Agentic systems run long, multi-turn loops that repeatedly reuse large prefixes (system prompts, memory, retrieved context, tool outputs) while adding only small increments of new tokens. Contexts can reach tens or hundreds of thousands of tokens, with input-to-output ratios exceeding 100:1. In theory, most of this work should be reused via KV caching.\n\nIn practice, reuse breaks down. Systems lack visibility into which instances hold relevant KV cache, memory pressure forces eviction of large prefixes, and requests often land on different instances across turns. This leads to repeating prefill, lost cache locality, degraded latency, and lower throughput, even when systems appear balanced.\n\nInfera addresses this by coordinating engine instances across three dimensions: KV-aware routing, prefill-decode disaggregation, and KV-cache tiering. Rather than replacing inference engines, it manages request placement, execution phases, and reusable KV state across the deployment.\n\nThis orchestration increases inference goodput: the rate of inference requests completed within latency targets such as time to first token and inter-token latency. This metric is essential for end-to-end agentic AI performance, where sequential inference calls, tool use, and expanding context compound latency.\n\n## Three Capabilities That Work Together[#](#three-capabilities-that-work-together)\n\nInfera helps make key decisions in three stages of the distributed inference request lifecycle, as shown in Figure 1:\n\n**Route: KV-Aware Routing:** Route the request to an engine instance that balances KV-cache reuse and active work.**Specialize: Prefill-Decode Disaggregation:** Run prefill and decode on specialized engine instances.**Retain: KV-Cache Tiering:** Retain reusable KV state beyond GPU HBM memory for future requests.\n\nFigure 1. Infera capabilities\n\n### Route: KV-Aware Routing[#](#route-kv-aware-routing)\n\nThe first decision is where a request should be run.\n\nConventional load balancing uses signals such as queue depth or utilization, but it does not account for work an engine instance has already performed. Agentic requests often share long prefixes, including system prompts, retrieved documents, source code, tool results, and growing conversation histories.\n\nRouting a request to an engine instance that already caches much of its prefix enables reuse. Routing the same request to a cold engine instance forces the prompt to be processed again.\n\nEngine instances publish KV-cache events, and the router maintains an in-memory view of the prompt blocks cached by each instance. For an incoming request, it estimates how many blocks each candidate would need to compute, combines that cost with the active prompt blocks already queued on the instance, and selects the lowest-cost option.\n\nThis balances locality and active work rather than optimizing either signal in isolation.\n\nThis approach increases KV-cache reuse, reduces redundant prefill computation, and lowers time to first token for requests with repeated prefixes, directly improving responsiveness and overall system efficiency.\n\n### Specialize: Prefill-Decode Disaggregation[#](#specialize-prefill-decode-disaggregation)\n\nThe next decision is whether to specialize inference engine instances for different phases of the workload.\n\nLLM inference has two phases with distinct characteristics. Prefill processes the prompt and builds the KV cache, making it primarily compute-bound. Decode generates tokens incrementally and is primarily memory-bandwidth-bound. Because the two phases stress the system differently, their scaling requirements can also differ.\n\nInfera separates prefill and decode into specialized instances when the benefits outweigh the KV-transfer cost; otherwise, both phases run in a single instance. With disaggregation, the prefill instance processes the prompt and transfers the resulting KV state to a compatible decode instance.\n\nAcross hosts, Infera uses an RDMA-capable transfer path such as Mooncake or MoRI-IO and validates the negotiated transport to avoid unintended TCP fallback and the resulting transfer latency.\n\nDisaggregation allows prefill and decode to operate and scale independently. Its benefits become most visible in larger deployments, where contention, concurrency, throughput, and latency must be managed against end-to-end service-level objectives. By isolating the two phases, Infera can improve latency predictability, sustain higher concurrency, and help more requests meet their SLO targets under load.\n\n### Retain: KV-Cache Tiering[#](#retain-kv-cache-tiering)\n\nThe final decision is how to preserve reusable state.\n\nGPU HBM memory is the fastest place to hold KV cache, but its capacity is limited. Long contexts and concurrent agents can consume that capacity quickly. Evicting older blocks creates room for new work, but it also discards computation that may soon be valuable again.\n\nInfera supports AMD Infinity Context (AIC), which extends effective KV-cache capacity by offloading valuable GPU HBM KV-cache data to local NVMe or remote NFS-backed storage. Through AIC’s direct GPU data path, KV-cache data can move between GPU memory and storage without being staged through CPU DRAM, avoiding unnecessary host-side copies and enabling substantially higher transfer bandwidth.\n\nThis reduces the amount of costly CPU memory required for KV-cache tiering while preserving reusable prompt state and improving prefill performance.\n\nBy extending effective cache capacity and maintaining high cache-hit rates, KV-cache tiering minimizes recomputation, stabilizes latency for long-context workloads, and improves overall goodput in agentic systems.\n\n## Built Around the Inference Engines You Already Use[#](#built-around-the-inference-engines-you-already-use)\n\nInfera works with existing inference engines rather than replacing them. It supports vLLM, SGLang, and the AMD ROCm-native ATOM engine. Each engine continues to own model execution within its engine instances, while Infera provides orchestration across them. This separation preserves existing tooling and engine-specific optimization while allowing the engines and Infera to evolve independently. The latest feature support for each engine is detailed in the [docs](https://rocm.docs.amd.com/infera).\n\n## Technical Architecture[#](#technical-architecture)\n\nInfera introduces a focused orchestration layer to the ROCm inference architecture stack, as shown in Figure 2.\n\nFigure 2. ROCm Infera architecture\n\nApplications connect to the Infera server through OpenAI-compatible APIs (with an Anthropic-compatible shim), where a built-in router handles scheduling. The server is designed for high concurrency: multiple replicas scale out behind a load balancer, so no single instance becomes a bottleneck, and a failed replica does not disrupt the overall service. For the most demanding routing paths, the same logic is also available as a high-performance Rust router that further increases throughput and reduces coordination overhead.\n\n### Infera Request Flow[#](#infera-request-flow)\n\nFigure 3 details the request flow between components at runtime, including how routing, external KV-cache lookup, prefill execution, KV transfer, and decode generation work together.\n\nFigure 3. The request flow in a prefill-decode disaggregated deployment with external KV-cache storage.\n\nBefore serving requests, the prefill and decode engine instances register with the control plane through etcd, enabling the router to discover their roles and capabilities. The request flow then proceeds as follows:\n\n**Receive the request.** A client sends a request to Infera through the router.**Select engine instances.** The router chooses a compatible prefill-decode pair based on role, active work, and KV-cache locality.**Check the external KV cache.** Before recomputing the prompt, the prefill instance queries kvd for reusable KV-cache blocks.**Restore cached state.** On a cache hit, AMD Infinity Context (AIC) restores the available KV-cache blocks from local NVMe or remote NFS through a direct GPU path. The prefill instance then computes only the uncached portion of the prompt.**Transfer the completed KV state.** After prefill completes, the KV cache is transferred to the decode instance through a compatible PD connector using Mooncake or MoRI-IO.**Generate and return the response.** The decode instance generates the output tokens and streams the response back to the client through the router.\n\nThe inference engines execute the model, while Infera coordinates service discovery, request routing, external KV-cache reuse, and the handoff between prefill and decode engine instances.\n\n## Preliminary Performance[#](#preliminary-performance)\n\nInfera v0.1 is an initial open-source release, and performance evaluation is ongoing. We evaluated a synthetic 1K-input/1K-output workload and a long-context agentic workload. The results below provide an early view of internal testing on AMD Instinct™ MI355X GPUs using Kimi K2.6 MXFP4 and are not a comprehensive characterization across models, workloads, engines, or deployment topologies.\n\nGoodput measures the inference work completed while meeting a defined responsiveness target. Unlike raw throughput, it excludes work delivered too slowly to provide the intended user experience. Figure 4 summarizes the relative goodput-per-GPU gains observed across the evaluated workloads.\n\n### Higher Goodput per GPU Across Workloads[#](#higher-goodput-per-gpu-across-workloads)\n\nFigure 4. Infera goodput per GPU relative to the corresponding single-instance baseline = 1.0×.\n\n#### Synthetic 1K/1K Workload[#](#synthetic-1k-1k-workload)\n\nOn the synthetic 1K/1K workload, an **Infera 20-GPU 1P2D configuration** delivered up to **2.7× higher goodput per GPU**[[1]](#id3) than a **single-instance** **4-GPU TP4 SGLang baseline** while maintaining **30 output tokens per second per user**.\n\nThe improvement comes primarily from prefill-decode disaggregation and Data Parallel Attention (DP Attention). Disaggregation separates prompt processing from token generation, allowing each phase to scale independently. DP Attention increases the aggregate GPU-resident KV-cache capacity across the serving pool. Together, these techniques support more concurrent requests while maintaining responsive token generation.\n\n#### Long-Context Agentic Workload[#](#long-context-agentic-workload)\n\nFor the long-context agentic workload, we evaluated a coding-agent workload using the multi-turn dataset from the [vLLM Mooncake Store blog](https://vllm.ai/blog/2026-05-06-mooncake-store) to represent long-running coding and tool-using agent sessions. Each session starts from a 20K-token shared prefix and runs up to 30 turns, adding ~2K input tokens and ~900 output tokens per turn, so the accumulated context reaches ~75K tokens at P50 and ~115K at its maximum.\n\nOn the long-context agentic workload, an **Infera 12-GPU prefill-decode (PD) configuration (prefill 4-GPU TP4, decode 8-GPU TP2×DP4)** delivered up to **1.7× higher goodput per GPU**[[2]](#id4) than a **single-instance 8-GPU TP8 vLLM baseline** while maintaining **30 output tokens per second per user**. At the stricter target of **50 output tokens per second per user**, Infera delivered up to **2.6× higher goodput per GPU**.\n\nAs concurrency increased, long-context prompt processing interfered with active token generation in the baseline, causing per-user performance to fall below the target. Infera separated the two phases: dedicated prefill instances processed the growing input context, while decode instances focused on generating output tokens.\n\nThese early results show how Infera converts higher concurrency into usable goodput rather than raw throughput alone. Detailed workload definitions, configurations, methodology, and performance reproduction instructions are available in the [benchmarks section of the repository](https://github.com/AMD-AGI/Infera/tree/main/examples). We are continuing to expand our test coverage.\n\n## Roadmap[#](#roadmap)\n\nInfera is currently at **v0.1**, and the initial reference solution should be viewed as a foundation rather than a feature-complete solution. Current limitations are detailed in the Infera [Feature Matrix](https://rocm.docs.amd.com/projects/infera/en/latest/features/feature_matrix.html) and [Compatibility Matrix](https://rocm.docs.amd.com/projects/infera/en/latest/features/compatibility_matrix.html). The public [Infera Roadmap](https://github.com/AMD-AGI/Infera/issues/9) identifies the following priorities.\n\n**Dynamic scaling under Kubernetes.** Add load-driven replica-count autoscaling. Runtime switching between prefill and decode roles is planned as a design and prototype.\n\n**SLO-aware scheduling.** Make TTFT and TPOT targets configurable and add SLO-attainment and goodput reporting.\n\n**Broader AMD Instinct support.** Expand validation beyond MI355X to MI300X, MI325X, and MI455X with clear documentation of validated and expected-to-work platforms.\n\n**Multimodal support.** Add multimodal-aware hashing and routing and explore encode-prefill-decode disaggregation.\n\n**Research directions.** Assess wide expert parallelism, cluster-wide KV-cache sharing across RDMA and NVMe, and xGMI or remote-copy transports for KV movement.\n\nScope and sequencing may evolve as the architecture matures, and the community provides feedback.\n\n## Summary[#](#summary)\n\nIn this blog, you learned why scaling agentic AI requires more than adding engine instances: it requires fleet-wide coordination to preserve cache locality, reduce interference between prefill and decode, and keep valuable context available for reuse.\n\nInfera is a conductor for your inference GPU orchestra, coordinating the instance fleet while leaving model execution to vLLM, SGLang, and ATOM. Future posts will go deeper into deployment patterns, benchmark methodology, and practical guidance for building higher-goodput agentic AI serving systems on AMD Instinct GPUs.\n\n## Get Started[#](#get-started)\n\n**Docs:**[rocm.docs.amd.com/infera](https://rocm.docs.amd.com/infera)\n\n## Disclaimers[#](#disclaimers)\n\nThe information presented in this document is for informational purposes only and may contain technical inaccuracies, omissions, and typographical errors. The information contained herein is subject to change and may be rendered inaccurate for many reasons, including but not limited to product and roadmap changes, component and motherboard version changes, new model and/or product releases, product differences between differing manufacturers, software changes, BIOS flashes, firmware upgrades, or the like. Any computer system has risks of security vulnerabilities that cannot be completely prevented or mitigated. AMD assumes no obligation to update or otherwise correct or revise this information. However, AMD reserves the right to revise this information and to make changes from time to time to the content hereof without obligation of AMD to notify any person of such revisions or changes. THIS INFORMATION IS PROVIDED ‘AS IS.” AMD MAKES NO REPRESENTATIONS OR WARRANTIES WITH RESPECT TO THE CONTENTS HEREOF AND ASSUMES NO RESPONSIBILITY FOR ANY INACCURACIES, ERRORS, OR OMISSIONS THAT MAY APPEAR IN THIS INFORMATION. AMD SPECIFICALLY DISCLAIMS ANY IMPLIED WARRANTIES OF NON-INFRINGEMENT, MERCHANTABILITY, OR FITNESS FOR ANY PARTICULAR PURPOSE. IN NO EVENT WILL AMD BE LIABLE TO ANY PERSON FOR ANY RELIANCE, DIRECT, INDIRECT, SPECIAL, OR OTHER CONSEQUENTIAL DAMAGES ARISING FROM THE USE OF ANY INFORMATION CONTAINED HEREIN, EVEN IF AMD IS EXPRESSLY ADVISED OF THE POSSIBILITY OF SUCH DAMAGES. AMD, the AMD Arrow logo, and combinations thereof are trademarks of Advanced Micro Devices, Inc. Other product names used in this publication are for identification purposes only and may be trademarks of their respective companies. © 2026 Advanced Micro Devices, Inc. All rights reserved\n\n### Cautionary Statement[#](#cautionary-statement)\n\nThis blog may contain forward-looking statements concerning Advanced Micro Devices, Inc. (AMD), which are made pursuant to the Safe Harbor provisions of the Private Securities Litigation Reform Act of 1995. Forward-looking statements are commonly identified by words such as “would,” “may,” “expects,” “believes,” “plans,” “intends,” “projects” and other terms with similar meaning. Investors are cautioned that any forward-looking statements in this blog are based on current beliefs, assumptions and expectations, speak only as of the date of this blog and involve risks and uncertainties that could cause actual results to differ materially from current expectations. Such statements are subject to certain known and unknown risks and uncertainties, many of which are difficult to predict and generally beyond AMD’s control, that could cause actual results and other future events to differ materially from those expressed in, or implied or projected by, the forward-looking information and statements. Investors are urged to review in detail the risks and uncertainties in AMD’s Securities and Exchange Commission filings, including but not limited to AMD’s most recent reports on Forms 10-K and 10-Q.\n\nAMD does not assume, and hereby disclaims, any obligation to update forward-looking statements made in this blog, except as may be required by law.", "url": "https://wpnews.pro/news/introducing-amd-rocm-infera-scaling-goodput-for-agentic-ai-with-distributed", "canonical_source": "https://rocm.blogs.amd.com/software-tools-optimization/infera-di/README.html", "published_at": "2026-07-21 00:00:00+00:00", "updated_at": "2026-07-21 21:56:28.950346+00:00", "lang": "en", "topics": ["artificial-intelligence", "ai-infrastructure", "ai-tools", "ai-products", "ai-research"], "entities": ["AMD", "AMD ROCm Infera", "AMD Instinct", "vLLM", "SGLang", "ATOM", "GitHub"], "alternates": {"html": "https://wpnews.pro/news/introducing-amd-rocm-infera-scaling-goodput-for-agentic-ai-with-distributed", "markdown": "https://wpnews.pro/news/introducing-amd-rocm-infera-scaling-goodput-for-agentic-ai-with-distributed.md", "text": "https://wpnews.pro/news/introducing-amd-rocm-infera-scaling-goodput-for-agentic-ai-with-distributed.txt", "jsonld": "https://wpnews.pro/news/introducing-amd-rocm-infera-scaling-goodput-for-agentic-ai-with-distributed.jsonld"}}