NVIDIA Vera Rubin and Blackwell Set a New Standard for Agentic AI Performance per Watt

SemiAnalysis's AgentX benchmark shows NVIDIA's Vera Rubin NVL72 delivers up to 30x higher AI-factory throughput per megawatt than GB300 NVL72 for agentic-coding inference, while GB300 NVL72 extends its order-of-magnitude advantage over prior generations. The benchmark replays production-style agent sessions to measure long-context prefill, KV-cache reuse, and interactive decode under realistic concurrency.

AI agents have expanded inference from single-turn interactions into multi-step workflows that reason, invoke tools, coordinate subagents, and carry growing context from one turn to the next. The scale of this shift is now visible in raw consumption: across 100 trillion tokens of real-world usage, OpenRouter’s State of AI https://openrouter.ai/state-of-ai report found that average prompt tokens per request grew roughly fourfold, and single agentic requests consume 15 times the tokens of ordinary chat. Properly characterizing hardware performance for this workload presents new challenges. A useful benchmark must capture long-context prefill, KV-cache reuse, interactive decode, tool-call gaps, and distributed mixture of experts MoE execution under realistic concurrency. It must also show how much of the AI-factory power budget is converted into useful agentic throughput while maintaining an acceptable user experience. This post examines SemiAnalysis AgentX https://inferencex.semianalysis.com/inference?g model=Kimi-K3&i seq=agentic-traces&i optimal=1 , a benchmark that evaluates AI infrastructure for agentic-coding inference using replayed production-style sessions. It highlights Vera Rubin NVL72 preview results showing up to 30x higher AI-factory throughput per megawatt than GB300 NVL72, while showing that Blackwell GB300 NVL72 extends its order-of-magnitude throughput-per-megawatt advantage over prior generations to dynamic agentic workloads. What is AgentX? AgentX https://inferencex.semianalysis.com/inference?g model=Kimi-K3&i seq=agentic-traces&i optimal=1 is the agentic-coding benchmark in InferenceX https://inferencex.semianalysis.com/ , SemiAnalysis’s open-source benchmark suite. It measures how efficiently accelerators serve the request patterns produced by real coding agents. AgentX testing and measurement methodology Agentic sessions are long, stateful, and variable: they chain model calls, tool use, and growing context rather than following a fixed prompt-and-response pattern Figure 1, below . AgentX therefore measures whether a platform can serve replayed agent traffic responsively, reuse previously processed context, and maximize agentic throughput per provisioned megawatt. Figure 2, below, shows the legacy InferenceX static 8K/1K sequence length results for DeepSeek-R1-0528. GB300 NVL72 leads H200 by up to 40x more tokens per megawatt. However, the scenario represents controlled, fixed-length serving rather than the traffic generated by an agent in production. In real agent sessions, request lengths vary from turn to turn. Context accumulates throughout a task, previously processed tokens can be reused, and model calls are interrupted by tool execution or delegated work. As agentic AI has become the leading workload, a fixed sequence length scenario has become less representative of true serving performance and has been demoted to “maintenance mode” on the InferenceX suite. AgentX addresses that gap by measuring serving performance across prerecorded Claude Code sessions https://inferencex.semianalysis.com/datasets with interleaved reasoning and tool use. It replays each session turn by turn using the AIPerf client https://github.com/SemiAnalysisAI/aiperf . Because every system receives the same recorded traffic, observed differences reflect the serving stack rather than benchmark-specific tuning. Figure 3, below, illustrates a sample pre-recorded session. The replay preserves each session’s context and input/output sequence lengths, as well as the reasoning time and tool-call latency captured in the original trajectory. These intervals preserve the original session’s timing, reproducing KV cache-capacity pressure that a realistic benchmark must capture. AgentX varies concurrency to map the trade-off between throughput and interactivity. At each operating point, it reports sustained throughput per provisioned megawatt alongside each stack’s ability to reuse repeated context How to interpret AgentX results The AgentX metric that matters most for AI factories is tokens per megawatt. The benchmark reports this metric against four user-experience values: E2E normalized interactivity, standard interactivity, E2E latency, and time to first token TTFT . Table 1, below, describes how to best leverage these. Metric | Definition | Usefulness | | E2E Normalized Interactivity | The average output-token rate per user over the full request, calculated by dividing total output tokens by the time from request submission to final-token delivery. | Shows how much user-visible output a platform delivers per megawatt for a given end-to-end experience, including time to first token. Higher is better. | | Standard Interactivity | The per-user token rate during generation alone, calculated as output tokens divided by the elapsed time from the first token to the last. | Shows how much streamed output a platform delivers per megawatt once generation begins, excluding time to first token. Higher is better. | | E2E Latency | The total elapsed time of a single request, measured from submission to the arrival of the final output token. | Tests whether a throughput-per-megawatt result is usable: high efficiency does not help if requests take too long to finish. Less is better. | | TTFT | The elapsed time from submission to the arrival of the first output token. | Shows whether power-efficient throughput is paired with a fast first response, which is critical when agents repeatedly begin long-context turns. Lower is better. | Table 1. Key AgentX performance metrics NVIDIA Vera Rubin NVL72 Results The following Vera Rubin NVL72 results were measured by NVIDIA using the SemiAnalysis AgentX workload and are pending SemiAnalysis review. As shown in Figure 4, below, at 160 tokens per second per user on the AgentX DeepSeek V4-Pro workload, Vera Rubin NVL72 delivers up to 30x higher AI-factory throughput per megawatt than GB300 NVL72, pointing to a substantial increase in agentic inference capacity while maintaining the same interactive serving target. To learn more about how NVIDIA Vera Rubin NVL72 was designed to deliver maximum performance per megawatt, see Inside NVIDIA Rubin GPU Architecture: Powering the Era of Agentic AI. https://developer.nvidia.com/blog/inside-nvidia-rubin-gpu-architecture-powering-the-era-of-agentic-ai/ NVIDIA GB300 NVL72 Results On AgentX, GB300 NVL72 delivers up to 15x higher AI-factory throughput per megawatt than H200 NVL8 for DeepSeek V4 Pro 1.6T on the AgentX workload see Figure 5, below . In other words, GB300 NVL72 sustains substantially more responsive agentic inference throughput within the same power budget. That throughput advantage translates directly into unit economics. GB300 NVL72 delivers up to 10x lower cost per million tokens than H200 NVL8 Figure 6, below . For operators, this means a fixed power and infrastructure budget can support materially more interactive agentic capacity, or deliver the same capacity with substantially lower operating cost. The advantage of GB300 NVL72 becomes more pronounced as model scale increases. Figure 7, below, shows throughput per megawatt for Kimi K3 2.8T on the AgentX workload. With roughly 80x the throughput per megawatt of H200 NVL8 at comparable interactivity, GB300 NVL72 also extends the interactivity frontier to roughly 215 tokens per second per user, well beyond the operating range reached by H200 NVL8. These GB300 NVL72 results reflect system-level work across the serving runtime, model kernels, and scale-up fabric. Together, these layers enable large MoE models to sustain responsive throughput as agent sessions accumulate context, concurrency increases, and decode demand intensifies. MoE serving runtimes: Frameworks including SGLang https://github.com/sgl-project/sglang , TensorRT-LLM https://github.com/NVIDIA/TensorRT-LLM , and vLLM https://github.com/vllm-project/vllm can distribute expert execution across the NVL72 domain. Techniques such as Wide Expert Parallelism and DeepEP help balance expert work across more GPUs, increasing the effective batch size available for concurrent agent requests. MoE kernels and communication overlap: DeepGEMM-based kernels, mixed-precision formats such as MXFP4 and MXFP8, and fused MoE execution paths reduce time spent moving data between expert stages. By overlapping expert-parallel communication with Tensor Core computation, the serving stack can improve token throughput for reasoning and coding workloads.Dynamo separates prefill and decode into independently scaled worker pools, allowing each phase to be configured for its own performance requirements. It also supports session-aware serving through session IDs that correlate related model calls, tool spans, and traces across an agent run. Separately, its KV-cache-aware router uses cache overlap and worker load to select a target, reducing unnecessary prefill computation and helping maintain responsive multi-turn serving as agent sessions reuse context. NVIDIA Dynamo https://github.com/ai-dynamo/dynamo :NVLink connects the 72 GPUs in GB300 NVL72 in a high-bandwidth scale-up domain. This enables the compute, memory, expert-parallel communication, and KV-cache movement required to serve large models as a coordinated rack-scale system. NVIDIA NVLink https://www.nvidia.com/en-us/data-center/nvlink/ scale-up fabric: What’s next for agentic AI? Vera Rubin NVL72 shows what is possible when a rack-scale system is tuned for the long-context, interactive, and distributed execution patterns of agentic inference. The broader Vera Rubin platform extends that approach across the full workflow. Rubin GPUs process large context and decode efficiently, Vera CPUs handle tool execution and KV-cache offload, and Groq 3 LPX unlocks ultrafast interactivity. Across the AI factory, NVLink 6, ConnectX-9, BlueField-4, and Spectrum-X move tokens, context, and tool results between resources. Dynamo, Attention-FFN Disaggregation, NVFP4, TensorRT-LLM WideEP, and speculative decoding coordinate execution across the most suitable processors. The goal is simple: reduce recomputation and waiting, sustain interactive performance as agent sessions grow, and convert more of a fixed power budget into useful agentic output. To explore the technologies and benchmarks behind this result, start with the resources below. - Explore the architecture and capabilities of NVIDIA Rubin https://developer.nvidia.com/blog/inside-the-nvidia-rubin-platform-six-new-chips-one-ai-supercomputer/ - Review live benchmark results on the SemiAnalysis InferenceX dashboard https://inferencex.semianalysis.com/inference?g model=Kimi-K3&i seq=agentic-traces&i optimal=1 - Learn how NVIDIA’s extreme co-design addresses the growing complexity, latency, and economics of agentic systems in Building for the Rising Complexity of Agentic Systems with Extreme Co-Design https://developer.nvidia.com/blog/building-for-the-rising-complexity-of-agentic-systems-with-extreme-co-design/ . Acknowledgments This work was made possible through the expertise and engineering contributions of Xin Li, Ankur Singh, Anthony Casagrande, Jonas Li, Po-Han Huang, Xiaoming Chen, and many other talented NVIDIA engineers.