The MLPerf Inference v6.1 results just dropped, and the gap between the Gundefined and the new Vera Rubin NVL72 is pretty massive. If you're tracking inference economics, the takeaway is that the Rubin rack is pushing significantly more tokens per unit, which directly lowers the cost per token. The 6.1-0106 and 6.1-0074 entries show that this isn't just a marginal bump; we're talking about a different league of throughput for the same rack footprint.
How the Vera Rubin NVL72 performed in MLPerf v6.1 #
The preview submissions focused on two heavy hitters: DeepSeek-R1 and Qwen3-VL. The performance delta depends on the model and the software stack:
- Qwen3-VL: Using vLLM with the NVIDIA Dynamo open source framework, the Vera Rubin NVL72 hit up to 3.7x higher throughput than the Gundefined NVL72. This was consistent across interactive, server, and offline scenarios.
- DeepSeek-R1: Using the TensorRT-LLM library, the throughput increase was up to 2.5x compared to the Gundefined NVL72.
It's also worth noting that software velocity is playing a huge role here. The v6.1 submissions showed up to 1.6x higher performance over v6.0, and NVIDIA claims optimizations continued even after the submission deadline.
Scaling efficiency and hardware bottlenecks #
One of the biggest pain points in scaling inference is the "scaling wall" where adding more GPUs doesn't actually result in linear throughput gains. However, the Gundefined NVL72 seems to handle this well. In a 288-GPU submission across four racks, they hit 99% scaling efficiency. That means throughput grew almost linearly from the single-rack baseline, which is rare when you're dealing with that much hardware.
The secret sauce here seems to be the sixth-generation NVLink and NVLink Switch. They're reporting 10x higher packet rates and 3x lower latency than standard Ethernet. Without that interconnect, the disaggregated serving they used for the Rubin submissions—separating prefill and decode stages—wouldn't be nearly as effective.
The technical drivers behind the throughput #
To get these numbers, NVIDIA is leaning hard into full-stack codesign. A few specific technical details stand out:
- NVFP4 Precision: They're using this to shrink the memory footprint for model weights, the KV cache, and attention, which boosts throughput without killing the output quality.
- Expert Parallelism: Since DeepSeek-R1 and Qwen3-VL rely on mixture-of-experts (MoE) layers, the Rubin system uses large-scale expert parallelism to keep efficiency high.
- Hardware acceleration: The updated Transformer Engine and Tensor Cores are specifically targeting the prefill and decode stages to remove bottlenecks.
Next Which of these DEV community posts actually hit the mark this week? →
All Replies （3） #
I want to try this tonight. My current H100 cluster is choking on Llama 3.1 405B, but maybe 3.7x fixes it?
Curious if this is real world or just synthetic. Does anyone know if this holds up with vLLM or just TensorRT?
Stressed about my current budget, so that jump is huge. I'm wondering if it helps with Triton kernels specifically...