Every AI roadmap I read still treats FLOPs as the scarce resource. They aren't. FLOPs are now the most abundant thing in the building. The constraint has already moved twice, and it's about to move again, out of the chip and into the power grid.
If you build AI products, this isn't trivia for hardware people. It decides what your inference costs, which architectures win, and which features you can afford to ship. Here's the argument from first principles, using numbers the vendors publish themselves.
Put Nvidia's current rack generation next to the one it replaces. The figures below come from Nvidia's Vera Rubin NVL72 spec page and a side-by-side compiled by Lightyard from Nvidia's datasheets. GB300 rack power comes from an OEM guide, since Nvidia doesn't print it.
| Per rack | GB300 NVL72 | Vera Rubin NVL72 | Growth |
|---|---|---|---|
| FP8 dense compute | 360 PFLOPS | 1,260 PFLOPS | 3.5× |
| HBM bandwidth | 576 TB/s | 1,400 TB/s | 2.4× |
| NVLink bandwidth | 130 TB/s | 216 TB/s | 1.7× |
| Rack power | 135 kW TDP | 187 kW (Max-Q) | 1.4× | Read the growth column from top to bottom. Each resource grows more slowly than the one above it. Compute triples, memory bandwidth more than doubles, the interconnect grows by two-thirds, and the power you're allowed to pull grows by less than half.
That ordering is the whole story. Chips are getting cheaper and more capable faster than you can feed them, connect them or power them. The bottleneck isn't a single wall. It's a queue of walls, and the industry is walking down it.
Start with the physics of generating a token.
During autoregressive decode, every weight in the model has to stream from HBM into the compute units for each forward pass. With FP8 weights that's one byte per parameter, and each byte does roughly two floating-point operations (a multiply and an add) for each sequence in the batch. So decode arithmetic intensity is about 2 × batch size FLOPs per byte, before you count the KV cache, which only adds bytes.
Now look at what the hardware wants. A single Rubin GPU is listed at 17.5 PFLOPS dense FP8 and 19.2 TB/s of HBM4 bandwidth (Nvidia). Divide one by the other and you get a ridge point of about 910 FLOPs per byte. On GB300 (per GPU: 5 PFLOPS FP8 dense, 8 TB/s) it was about 625.
Two conclusions fall straight out of that division:
A quick sanity bound: a 70B-parameter model in FP8 is about 70 GB of weights. At 19.2 TB/s, one GPU can stream those weights at most about 270 times a second. That's the hard ceiling on batch-1 tokens per second for that model on that chip, no matter how many PFLOPS the marketing slide shows. In practice you'll land below it.
This is why the inference stack of 2026 looks the way it does. Every technique that matters is a bandwidth trick wearing a different hat:
If your team talks about "model quality" and never about bytes moved per token, it's optimizing the wrong variable. Once a model, its KV cache and its parallelism strategy no longer fit inside one GPU, every token turns into a communication problem. Tensor parallelism means all-reduce traffic on every layer. Expert parallelism means all-to-all traffic. Long-context serving means cache shuffling.
That's why the NVLink domain, not the GPU, has become the real unit of compute. A Rubin NVL72 rack is designed as one 72-GPU machine with 216 TB/s of NVLink, while its scale-out networking is listed at 32.4 TB/s per rack (Nvidia). That's roughly a 7× cliff between talking inside the rack and talking outside it.
The architectural rule follows from that cliff: keep the chatty traffic inside the domain and send only the polite traffic across the network. Tensor and expert parallelism live inside the rack. Data parallelism and pipeline stages cross it. Model designs that respect that boundary run well, and designs that don't burn money waiting on the wire.
For anyone renting rather than building, the practical translation is short. When a provider quotes "GPUs," ask about the domain size and the scale-out bandwidth. Two clusters with the same GPU count can differ several-fold in useful throughput on a large MoE model. Now zoom out to the building.
Nvidia's own reference design puts about 40,000 Rubin GPUs in a 100 MW AI factory (Nvidia). That's roughly 2.5 kW of facility power per GPU once cooling and networking are counted. The chip is no longer the expensive, scarce input. The megawatt is.
The macro numbers agree. The IEA estimates data centers used about 415 TWh in 2024 and projects about 945 TWh by 2030 in its base case, slightly more than Japan's entire electricity consumption today (IEA, Energy and AI). It projects electricity use by accelerated servers growing about 30% a year.
And here's the part most AI strategy decks skip. In the US, the median power project that came online in 2025 took 61 months from interconnection request to commercial operation, up from 36 months in 2015 and 22 in 2008. About 2,061 GW of capacity was still waiting in queues at the end of 2025 (Berkeley Lab, Queued Up 2026).
Put those two clocks side by side. GPUs ship on a roughly annual cadence. New grid power takes about five years. When one input improves every 12 months and another takes 60, the slow one sets the price of everything.
That's why Nvidia now publishes three power profiles for the same rack (Max-P at 227 kW, Max-Q at 187 kW and MaxLPS at 156 kW), and describes MaxLPS as a way "to maximize AI factory output within a fixed data center power budget" (Nvidia Vera Rubin datasheet). When the world's leading chip company sells you the option to run its chips slower so you can fit more of them under your power cap, it's telling you what the binding constraint is.
Here's the mental model I use. Call it the bottleneck ladder:
Two properties make the ladder useful:
Most companies will never buy a rack, but the ladder still sets their costs. Here's what I'd tell a CTO:
Price your product in joules, not tokens. Your provider's price per token is a lagging proxy for its power and bandwidth costs. Features that need long contexts, high reasoning budgets or batch-1 latency are expensive because they fight rung 2 and rung 4. Know which of your features do that.
Make latency a product decision, not a default. Batch-friendly workloads such as overnight analysis, async agents and bulk extraction ride the cheap side of the ridge. Real-time chat rides the expensive side. If a workflow can tolerate 30 seconds, design it to, and buy batch pricing where it exists.
Cut tokens before you cut models. Retrieval that sends 4,000 relevant tokens instead of 40,000 "just in case" tokens is a 10× change in bytes moved. That beats any vendor negotiation you'll ever have.
Route by difficulty. A small model drafting and a large model verifying is speculative decoding at the application layer. Most traffic in most products is easy. Pay frontier prices only for frontier problems.
Treat "AI compute is getting cheaper" with care. The price of a fixed capability falls fast. The price of frontier capability, served at low latency with long context, is pinned to the slowest rung on the ladder. Plan budgets for both curves.
The last decade of AI was a race for FLOPs. This decade is a race for watts, bytes and wires. The winners will move the fewest bytes, per token, per joule, inside the smallest communication domain that fits the job.
The chip isn't the product anymore. The power-constrained system is. Design accordingly.
Michael, ArkNet Digital: AI systems, agent infrastructure and security engineering. arknet.digital
If you're sizing inference costs, or deciding between real-time and batch architectures for an AI feature, I'm glad to compare notes. Disagree with the ladder? Tell me which rung I've got wrong in the comments.