Photo: Tara Winstead / Pexels
A new benchmark from Epoch AI measures chips by how many users they can serve, and peak FLOPs look less useful as a result
Epoch AI has introduced a new way to grade AI chips, and Nvidia’s newest hardware comes out of it looking very good. The research group’s Inference Performance Index, unveiled on November 9, 2026, finds that Nvidia’s Blackwell Ultra-based GB300 beats the H100 by a factor of 10 on inference.
What the Inference Performance Index actually measures #
Epoch AI’s index combines three ingredients: compute, memory bandwidth, and communication between chips.
The output is framed around a practical question. How many people can use the hardware at the same time while each still gets responses at a fixed output speed?
That framing matters because inference, the stage where a trained model actually answers prompts, is where AI models meet paying users. Training happens once. Inference happens every time someone types a question.
Why the GB300 pulls so far ahead #
The headline result is the tenfold inference advantage the index assigns to the GB300 over the H100, Nvidia’s Hopper-generation workhorse.
According to Epoch AI’s findings, the result reinforces a concern that FLOP/s figures undersell newer chip designs on the workloads that matter most for AI today.
Memory is a big part of the story. GB300 systems carry 288 GB of HBM3e memory per GPU and about 8 TB/s of memory bandwidth. The H100 offers roughly 3.35 TB/s.
That works out to approximately 2.4 times the memory bandwidth.
AI, tech, and the markets they move—in one daily briefing.
Daily. Free. Join 34,000+ readers across crypto, finance, and policy.
The GB300 also typically ships as part of the GB300 NVL72, a rack that links 72 GPUs together with upgraded NVLink interconnects. At the system level, that rack supports pooled memory of up to approximately 20–40 TB.
The design reduces how much a model needs to be split across chips through tensor parallelism. It also leans on lower-precision math tuned for inference workloads.
The rack-scale numbers reported elsewhere point in the same direction as Epoch AI’s index. The GB300 NVL72 reportedly delivers a 5–20× improvement in throughput, cost-per-token, and concurrency compared to the H100 across several AI models, including DeepSeek R1 and other large mixture-of-experts architectures.
MLPerf tests showed 5–7× advantages for the GB300 NVL72 over Hopper platforms on throughput and total cost of ownership.
Nvidia claims the GB300 NVL72 achieves up to 10× tokens per second per user and 5× tokens per second per megawatt against Hopper at rack scale.
The shift from chips to systems #
Epoch AI’s index fits a broader pattern that has played out across 2025 and 2026. AI hardware progress has increasingly come from integrated systems and optimized infrastructure, rather than from single-GPU specifications alone.
The GB300 NVL72 is a clear example. It is sold less as a chip and more as a rack-sized machine where memory, networking, and compute are engineered together.
By weighting memory bandwidth and chip-to-chip communication alongside raw compute, the Inference Performance Index rewards architectures that are good at the unglamorous plumbing.
What this means for buyers, builders, and Nvidia #
For Nvidia, the result is a tidy piece of third-party validation. The company has spent the past generation pitching rack-scale systems over standalone chips, and a metric that rewards memory and interconnect plays directly to that strategy. There are caveats. Any index reflects the choices its designers made about what to weigh and how, and different workloads stress hardware in different ways. The 5–20× range reported across models is itself a reminder that results depend heavily on what you run.
Disclosure: This article was edited by Diego Almada Lopez. For more information on how we create and review content, see our Editorial Policy.