SPARE: Fast Linear Attention Decoding with Compact Recurrent States

A training-free method called SPARE (State Projection for Accelerated Recurrent Execution) delivers a 2.55x full-model decoding speedup for linear attention by storing and updating recurrent states in fixed low-dimensional subspaces, according to the arXiv paper 2611.07552v1. SPARE also cuts recurrent-kernel DRAM traffic 7.49x and peak GPU memory 3.31x, while preserving better associative-retrieval accuracy than structured pruning at matched memory budgets. The method targets the quadratic scaling of recurrent states with representation dimension and the repeated data transfers that create a GPU memory bottleneck during decoding, using heterogeneous rank allocation and lightweight write compensation without online decomposition or dense-state reconstruction.

arXiv:2611.07552v1 Announce Type: new Abstract: Linear attention reduces the quadratic sequence-length complexity of softmax attention by maintaining fixed-size recurrent states. However, these states scale quadratically with the representation dimension and require repeated data transfers during decoding, creating a substantial GPU memory bottleneck. Recent studies have revealed significant low-rank redundancy in recurrent states, suggesting opportunities for more efficient representations. We introduce SPARE State Projection for Accelerated Recurrent Execution , a training-free method that exploits this redundancy through fixed low-dimensional subspaces, heterogeneous rank allocation, and lightweight write compensation. By performing recurrent reads and updates directly on compact states, SPARE reduces memory traffic without online decomposition or dense-state reconstruction. SPARE achieves 2.55x full-model decoding speedup, 7.49x lower recurrent-kernel DRAM traffic, and 3.31x lower peak GPU memory, while preserving better associative-retrieval accuracy than structured pruning at matched memory budgets.