Faster Than Flash: Exploiting Attention Sparsity for Efficient Long-Context Decoding

Researchers have introduced Faster Flash Decoding (FFD), a hardware-algorithm co-design framework that achieves up to 11.6x kernel-level speedup and 2.37x end-to-end throughput improvement for long-context LLM decoding, scaling to 256K context length while maintaining model accuracy on RULER and LongBench benchmarks. The training-free, plug-and-play method uses content-aware scanning via low-bit quantization and a top-delta strategy to exploit attention sparsity, with code available on GitHub.

arXiv:2609.00097v1 Announce Type: new Abstract: The development of long-context Large Language Models LLMs is constrained by the memory bandwidth bottleneck and quadratic complexity of the attention mechanism during decoding. To overcome the inherent trade-offs between the memory overhead of metadata-based metrics and the computational inefficiency of adaptive selection strategies, we present Faster Flash Decoding FFD , a novel hardware-algorithm co-design framework designed to break the memory wall in long-context decoding. FFD integrates the selector and computer into a fully fused kernel, replacing external metadata indices with content-aware scanning via low-bit quantization. Furthermore, we introduce the top-delta strategy, which dynamically filters blocks to achieve distribution-adaptive sparsity without global synchronization. Offering a training-free and plug-and-play solution, FFD also enables the reuse of scanning results for computation, achieving up to 11.6x kernel-level speedup and scaling to 256K context length, with 2.37x end-to-end throughput improvement. Empirical validation on RULER and LongBench confirms that FFD maintains model accuracy while delivering high-ratio sparsity, with code available at https://github.com/qluoluo/faster-flash-decoding