cd /news/ai-infrastructure/nvidia-vera-rubin-nvl72-leads-agenti… · home › topics › ai-infrastructure › article
[ARTICLE · art-170713] src=developer.nvidia.com ↗ pub= topic=ai-infrastructure verified=true sentiment=↑ positive

NVIDIA Vera Rubin NVL72 Leads Agentic Inference Performance on MLPerf Endpoints v1.0

NVIDIA reported provisional MLPerf Endpoints v1.0 results showing its Vera Rubin NVL72 delivers more than 10x the throughput per kilowatt of provisioned power of NVIDIA GB300 NVL72 at a shared minimum end-to-end interactivity of 50 tokens per second per user. In the Kimi K3 run published November 9, 2026, Vera Rubin NVL72 (32 accelerators) reached 41,414.2 system tokens/sec at 100.89 kW with 1,024 concurrent queries and a TTFT P90 of 4,673.9 ms, versus GB300 NVL72 (48 accelerators) at 2,210.4 tokens/sec and 60 kW with 48 queries. The Vera Rubin figures are pending MLCommons Association verification and peer review, while the GB300 results were verified by MLCommons.

by read7 min views4 publishedNov 16, 2026
NVIDIA Vera Rubin NVL72 Leads Agentic Inference Performance on MLPerf Endpoints v1.0
Image: NVIDIA Developer Blog

Agentic workloads are demanding more from inference systems. AI agents carry growing conversation histories across turns and generate a mix of reasoning, tool calls, and responses. Inference systems must support users at scale while remaining responsive.

MLPerf Endpoints v1.0 evaluates this balance between serving capacity and responsiveness using coding and enterprise workflows through API endpoints. During performance runs, generated tool calls and workflow intent are checked against reference values. Separate SWE-bench Verified evaluations assess end-to-end task-solving capability. These checks evaluate whether faster serving preserves model quality.

In MLPerf Endpoints v1.0, provisional NVIDIA Vera Rubin NVL72 results show over 10x the throughput per kilowatt of provisioned power of NVIDIA GB300 NVL72 at a shared minimum end-to-end interactivity of 50 tokens per second per user.<sup>1</sup>

This post presents the benchmark results and outlines the hardware and software optimizations used in the submissions.

<sup>1</sup>MLPerf® Endpoints v1.0, Kimi K3, Standardized division, Available category, published November 9, 2026. NVIDIA Vera Rubin NVL72, 32 accelerators, result 1.0.1.kimi-k3.agentic_combined.1p, run 9: concurrency = 1,024 queries, TTFT P90 = 4,673.9 ms, system TPS = 41,414.2 tokens/sec, E2E average interactivity = 50.4 tokens/sec per user, operating power = 100.89 kW. NVIDIA GB300 NVL72, 48 accelerators, result 1.0.0.kimi-k3.agentic_combined.1, run 6: concurrency = 48 queries, TTFT P90 = 2,305.1 ms, system TPS = 2,210.4 tokens/sec, E2E average interactivity = 56.2 tokens/sec per user, operating power = 60 kW. Comparison based on throughput per kilowatt at a shared minimum E2E average interactivity of 50 tokens/sec per user. Retrieved from endpoints.mlcommons.org on November 9, 2026. GB300 results verified by MLCommons Association. For the provisional Vera Rubin results: pending MLCommons Association verification, results are subject to change pending peer review. The MLPerf name and logo are registered and unregistered trademarks of MLCommons Association in the United States and other countries. All rights reserved. Unauthorized use is strictly prohibited. See www.mlcommons.org for more information.

The serving demands of multi-turn agentic AI #

The MLPerf Endpoints v1.0 dataset captures these workload characteristics: coding trajectories accumulate history over many turns, while enterprise workflows reuse shared instructions, tool definitions, and business rules. Figure 1 shows how context length grows across model responses during extended agentic sessions.

Vera Rubin delivers over 10x the agentic throughput per kilowatt #

The submissions span Kimi K3, Qwen3.6-35B-A3B, and DeepSeek-V4.1-Flash. Table 1 compares provisional Vera Rubin NVL72 results with verified NVIDIA GB300 NVL72 results for throughput per provisioned kilowatt at a shared minimum end-to-end (E2E) interactivity threshold and for E2E interactivity at matched query concurrency.

Model Throughput per provisioned kW ratio E2E interactivity ratio
Kimi K3 10x 3x
| DeepSeek-V4.1-Flash | 9x | 5x | 
| Qwen3.6-35B-A3B | 5x | 3x | 

Table 1. Throughput per kilowatt and E2E interactivity ratios for Vera Rubin NVL72 compared with GB300 NVL72²

² MLPerf® Endpoints v1.0, Kimi K3, DeepSeek-V4.1-Flash, and Qwen3.6-35B-A3B, Standardized division, Available category. GB300 results published November 4, 2026; Vera Rubin results published November 9, 2026, for Kimi K3 and Qwen, and November 10, 2026, for DeepSeek. Retrieved from

endpoints.mlcommons.orgon November 10, 2026. Comparisons select each platform’s best TPS/kW meeting the stated minimum E2E average interactivity, or best E2E average interactivity at the stated matched concurrency. Records list: result ID; run number; concurrency (queries); TTFT P90 (ms); System TPS (tokens/sec); E2E average interactivity (tokens/sec per user); operating-point power (kW).

Kimi K3: Throughput/kW: see Footnote 1. Interactivity at matched concurrency 1,024 queries: Vera Rubin (1.0.1.kimi-k3.agentic_combined.1p; 9; 1,024; 4,673.9; 41,414.2; 50.4; 100.89); GB300 (1.0.0.kimi-k3.agentic_combined.1; 16; 1,024; 15,223.0; 13,905.8; 16.8; 60).

DeepSeek-V4.1-Flash: Throughput/kW at minimum E2E average interactivity 280 tokens/sec per user: Vera Rubin (1.0.1.deepseek-v4_1-flash.mlperf_agentic_inference_dataset.1p; 6; 1,408; 427; 272,711.7; 294.49; 75.67), 24 GPUs; GB300 (1.0.0.deepseek-v4_1-flash.mlperf_agentic_inference_dataset.1; 1; 16; 112.8; 3,054.0; 288.65; 7.50), 4 GPUs. Interactivity at matched concurrency 1,024 queries: Vera Rubin (1.0.1.deepseek-v4_1-flash.mlperf_agentic_inference_dataset.1p; 5; 1,024; 505; 204,207.7; 307.36; 63.06), 20 GPUs; GB300 (1.0.0.deepseek-v4_1-flash.mlperf_agentic_inference_dataset.1; 8; 1,024; 4,073.5; 49,701.8; 56.65; 7.50), 4 GPUs.

Qwen3.6-35B-A3B: Throughput/kW at minimum E2E average interactivity 410 tokens/sec per user: Vera Rubin (1.0.1.qwen3_6-35b-a3b.agentic_coding.1p; 8; 320; 75.2; 47,785.5; 411.9; 12.61); GB300 (1.0.0.qwen3_6-35b-a3b.agentic_coding.1; 2; 36; 65.3; 5,960.4; 531.6; 7.50). Interactivity at matched concurrency 256 queries: Vera Rubin (1.0.1.qwen3_6-35b-a3b.agentic_coding.1p; 7; 256; 65.9; 41,142.6; 501.0; 12.61); GB300 (1.0.0.qwen3_6-35b-a3b.agentic_coding.1; 7; 256; 111.7; 23,467.8; 154.5; 7.50).

Ratios are rounded as shown. GB300 results verified by MLCommons Association. Vera Rubin’s results are provisional. Pending MLCommons Association verification, results are subject to change pending peer review. The MLPerf name and logo are registered and unregistered trademarks of MLCommons Association in the United States and other countries. All rights reserved. Unauthorized use is strictly prohibited. See

www.mlcommons.orgfor more information. Figure 2 shows the tradeoff between throughput per provisioned kilowatt and E2E interactivity across at least seven concurrency levels per curve. The curves show how much throughput each system delivers at a given responsiveness requirement. The step-function plot does not interpolate between measured points to keep comparisons grounded in demonstrated performance rather than estimates.

Higher throughput per kilowatt enables organizations to run more agents simultaneously within the same power budget while meeting the same responsiveness requirement. At low concurrency, faster responses reduce inference wait times for latency-sensitive agentic tasks.

Hardware and software optimizations for agentic inference #

NVIDIA Rubin GPUs combine HBM4 memory, NVIDIA Tensor Cores, and sixth-generation NVIDIA NVLink to support the demands of agentic inference. HBM4 enables rapid access to model weights and conversation state, Tensor Cores accelerate matrix operations, and NVLink supports communication across GPUs. Context-aware serving and model-specific kernels build on these capabilities to reuse prior work, independently optimize prefill and decode, and accelerate token generation—translating hardware advances into greater inference capacity and faster interactions.

Context-aware serving and model-specific optimizations improve inference performance

Agentic workloads repeatedly reuse conversation history while adding new instructions and tool results. Efficient serving must preserve reusable state, distribute requests across workers, and accommodate the different execution requirements of prompt processing and token generation. Prefix caching, state reuse, and NVIDIA Dynamo routing help reduce redundant computation and balance serving load, while disaggregated serving supports independent optimization of prefill and decode.

Figure 3 summarizes how context-aware serving and model-specific GPU execution support agentic inference.

Kimi K3, DeepSeek-V4.1-Flash, and Qwen3.6-35B-A3B also have distinct architectures that benefit from targeted GPU execution improvements. Model-specific kernels, fusion, optimized memory access, and speculative verification reduce data movement and execution overhead. Table 2 summarizes how these serving and execution techniques work together across the three models.

Optimization How it improves performance
Context reuse and routing Reuse previously processed conversation state and distribute requests to reduce recomputation and balance worker load. Example: Qwen’s fine-grained prefix matching reuses previously processed history across turns.
Prefill and decode optimization Configure prompt processing and token generation independently through disaggregated serving, reducing contention and synchronization waits. Example: Kimi combines disaggregated serving with prefill changes that reduce GPU synchronization waits.
Model-specific GPU execution Tailor kernels, fusion, and memory access to each architecture, reducing kernel launches and data movement. Example: DeepSeek’s MegaMoE kernel uses Rubin GPU locality and matrix-multiply instructions.
Speculative generation and verification Propose multiple tokens and verify them together, reducing sequential decoding steps. Example: Qwen’s multi-token prediction pairs token proposals with Tensor Core verification.

Table 2. Software optimizations improve context reuse, serving efficiency, and GPU execution

More productive infrastructure for agentic AI #

Agentic inference performance depends on co-design across hardware, system architecture, and serving software. The provisional MLPerf Endpoints v1.0 results show how the Vera Rubin platform brings these capabilities together to address the demands of agentic workloads and create more capacity to serve users and scale applications.

Learn more about NVIDIA training and inference performance in the companion blog post.

Acknowledgements #

Richard Cai, Viraat Chandra, Xiaoming Chen, Zhenhuan Chen, Alice Cheng, Julien Darve, Ashutosh Dhar, Kefeng Duan, Alec Flowers, Ann Guan, Jie Hao, Rashid Kaleem, Tzu-Ling Kan, Tuhin Khare, Zihao Kong, Jonas Li, Shicheng Li, Yijing Li, Allison Lim, Yanchao Lu, Bofeng Luo, Mingyuan Ma, Matthias Maiterth, Yibai Meng, Stanley Cheong Kwan Phoong, Hanjie Qiu, Zhongdao Ren, Tong Tong, Harshil Vagadia, Lisa Wang, Shang Wang, Xinyu Wang, Fernando Xiong, Yingying Xue, Yuanjing Xue, Lizhi Zhou, and Yaoyang Zhou

── more in #ai-infrastructure 4 stories · sorted by recency
── more on @nvidia 3 stories trending now
sponsored brought to you by zahid.host 4,200+ EU-deployed projects
reading about agents? ship yours in a single git push.

Run your AI side-project on zahid.host

EU-based hosting, git-push deploys, automatic HTTPS, no cold starts. Free tier with a custom domain — perfect for shipping the agent you just read about.

$git push zahid main
→ Live at https://your-agent.zahid.host ✓
Get free account → Pricing
from €0/mo · no card required
LIVE [news/nvidia-vera-rubin-nv…] indexed:0 read:7min 2026-11-16 · —