STEEL introduces a groundbreaking approach to running large language models on NPUs in laptops, achieving significant energy and speed advantages.
As large language models become integral to our daily tech interactions, the need for energy-efficient solutions is more pressing than ever. The rising trend of integrating these models in operating system workflows demands innovation, especially on laptop-class systems-on-chip (SoCs).
Breaking New Ground with STEEL #
STEEL emerges as a pioneering solution, marking the first open-source implementation of FlashAttention optimized for XDNA-like neural processing units (NPUs). The paper's key contribution: a novel dataflow formulation of prefill attention that capitalizes on spatial parallelism and on-chip memory. This enables a leap in efficiency.
Why does this matter? Cloud off, while popular, brings unwanted reliability and privacy issues. By shifting the heavy lifting to NPUs, STEEL addresses these concerns head-on. But, mapping attention mechanisms to NPUs isn't straightforward, owing to diverse architectures and explicit programming models.
Performance and Efficiency Gains #
On the performance front, STEEL doesn't disappoint. Tested on AMD's Ryzen AI 9 HX 370 SoC, it dwarfs traditional CPU and GPU implementations, slashing energy use by 9.17x and 1.75x, respectively. That's no small feat.
But the real showstopper? On XDNA 1, STEEL reduces latency by an average of 9.6x over prior SOTA, while on XDNA 2, it achieves a staggering 22.8x speedup compared to layer-by-layer attention implementations.
Implications for the Future #
This development raises a important question: Are NPUs the future of on-device AI processing? STEEL's success suggests we're heading in that direction. The ablation study reveals that optimizing for architectural specifics can lead to substantial gains in both speed and energy efficiency.
What's missing, though, is broader adoption across varying device types. For STEEL, or similar frameworks, to become a staple, more devices need to embrace these tailored NPU architectures. Will manufacturers take the bait? The potential benefits are clear.
Code and data are available at the project's repository, offering a transparent and reproducible pathway for others to follow or build upon. This builds on prior work from the field but takes it a step further with practical, demonstrable advancements.
Get AI news in your inbox
Daily digest of what matters in AI.