Via 9to5google.com
Turing Award winner David Patterson co-authors research arguing that AI's real bottleneck isn't computing power, it's memory
A new paper from Google DeepMind makes a case that should make every GPU-obsessed AI company : the biggest obstacle to scaling large language model inference isn’t raw computing power. It’s memory.
The research, authored by Xiaoyu Ma and Turing Award winner David Patterson, argues that the decode phase of autoregressive LLMs is fundamentally memory-bound and interconnect-limited. Translation: the most expensive chips in the world are sitting idle, waiting for data to arrive from memory, while the meter keeps running on your cloud bill.
The inference problem nobody wants to talk about #
When you ask ChatGPT a question, the model generates its response one token at a time. That sequential decode phase behaves very differently from the initial processing of your prompt (the prefill phase) or from training, both of which can exploit massive parallelism across thousands of cores. Decoding, by contrast, spends most of its time fetching key-value cache data from memory rather than doing math.
The paper, titled “Challenges and Research Directions for Large Language Model Inference Hardware” and submitted to arXiv in January 2026, identifies several trends making this problem worse. Mixture-of-Experts (MoE) models with up to 256 experts, long reasoning chains, extended context windows, multimodal inputs, and retrieval-augmented generation (RAG) all pile additional pressure on memory capacity, bandwidth, and latency.
Despite inference hardware sales projected to grow 4 to 6 times, the authors warn that ballooning service costs could jeopardize the economic sustainability of large-scale inference.
High Bandwidth Flash: the proposed fix #
To address these constraints, the paper proposes several hardware research directions. The most eye-catching is High Bandwidth Flash, or HBF, a memory technology designed to deliver roughly 10 times the capacity of current high-bandwidth memory solutions.
HBF aims for approximately 512 GB of capacity, compared to about 48 GB for HBM4, the next generation of conventional high-bandwidth memory. That dramatic expansion would allow much larger key-value caches to reside close to the compute units instead of requiring expensive, latency-inducing fetches across the datacenter network.
The technology would also need to address the known drawbacks of flash-based memory, including write endurance limitations and page-read latencies. The paper acknowledges these challenges directly, framing HBF as a research direction rather than a finished product.
Beyond memory, Ma and Patterson call for rethinking datacenter network designs and interconnect topologies, arguing that traditional network architectures weren’t designed for the communication patterns that modern inference workloads produce.
Rethinking how we measure AI hardware #
Perhaps the paper’s most provocative argument is that the industry is measuring success with the wrong yardstick. Peak FLOPS, the standard benchmark for accelerator performance, tells you how fast a chip can do math. But if the chip spends most of its time during inference waiting on memory, peak FLOPS is about as useful as measuring a restaurant’s quality by the horsepower of its delivery trucks.
The authors advocate for performance metrics centered on user-facing latency and total cost of ownership, the measures that actually determine whether an AI service makes money or burns it.
Patterson’s involvement lends significant weight to these arguments. As the co-creator of RISC architecture and a recipient of the ACM Turing Award, widely considered the Nobel Prize of computing, his assessments of hardware trends tend to shape industry direction. The paper has been accepted for publication in IEEE Computer magazine.
Disclosure: This article was edited by Editorial Team. For more information on how we create and review content, see our