Most people assume AI speed is all about raw horsepower—more TOPS or a faster NPU. But the real bottleneck is the "memory wall." While compute power has exploded, memory bandwidth hasn't kept pace. We spend more energy and time moving data from memory to the processor than actually calculating. This is especially brutal during the "Decode" phase of generative AI; every single token generated requires the model to read parameters and context. If the "chef" (compute) is fast but the "server" (bandwidth) is slow, the system just sits there waiting.
How High Bandwidth Compute (HBC) solves the data bottleneck #
Qualcomm is tackling this with High Bandwidth Compute (HBC), a near-memory computing architecture. Instead of dragging massive amounts of data across the chip, HBC moves the computation closer to the memory.
It is a middle ground between two other extremes:
- SRAM: Extremely fast but too expensive and small to fit today's massive LLMs.
- HBM: High capacity and fast, but involves complex stacking and packaging that makes it prohibitively expensive for most phones.
HBC doesn't replace everything. High-intensity tasks like "Prefill" (where the AI reads a long PDF) still need heavy-duty compute units. However, tasks like matrix multiplication, embedding queries, and KV Cache operations—which have high data-transport costs—are perfect for HBC. It is a strategic division of labor.
Optimizing the SoC for AI Agents #
An AI Agent phone is different from a basic chatbot. It has to jump between apps, read calendars, check weather, and recall chat history simultaneously. This "jumping" behavior is why Qualcomm is redesigning the layout of the next Snapdragon flagship.
The new Qualcomm Oryon CPU hits a 5GHz peak clock speed, making it the fastest mobile CPU globally. But the more interesting part is the Qualcomm Oryon FlexCache. This extensible cache allows different cores to share a pool of memory and dynamically grab more space when a heavy load hits. This means when an Agent switches tasks, the data stays in the cache instead of having to be fetched from system memory again.
Integrating AI into the Graphics Pipeline #
The next-generation Adreno GPU now includes Adreno Matrix Cores specifically for AI and matrix math. We are seeing a shift where things like frame generation and super-resolution are handled by neural networks rather than traditional algorithms.
To prevent the "memory wall" from hitting the GPU, Qualcomm added 18MB of Adreno High Performance Memory (HPM). This dedicated high-speed VRAM stores render targets and textures internally, so the GPU doesn't have to constantly communicate with the main system memory.
Expanding the NPU's working memory #
The new Hexagon NPU introduces the Element Accelerator, built specifically for Transformer workloads. More importantly, it has expanded its shared memory by up to 50%.
This is critical for the KV-cache (the AI's "working memory"). As a conversation gets longer, the context grows. By keeping that data closer to the accelerator, the NPU avoids frequent, slow trips to the DDR memory.
Finally, Qualcomm is implementing a Mixture-of-Experts (MoE) architecture. This allows a 30B parameter model to function while only activating about 3 billion parameters per token. This dynamic routing significantly lowers the actual compute load and the demand on memory bandwidth.
All Replies （3） #
Finally! My current Snapdragon 8 Gen 2 chokes on Llama 3. I wonder if this actually solves the 40ms latency.
Frustrated with my current setup since LLMs just crawl. I'm curious if this actually fixes the 12GB RAM bottleneck.
Excited to see this move. Will this actually impact the LPDDR5x speeds or is it just a cache change?