{"slug": "spare-fast-linear-attention-decoding-with-compact-recurrent-states", "title": "SPARE: Fast Linear Attention Decoding with Compact Recurrent States", "summary": "A training-free method called SPARE (State Projection for Accelerated Recurrent Execution) delivers a 2.55x full-model decoding speedup for linear attention by storing and updating recurrent states in fixed low-dimensional subspaces, according to the arXiv paper 2611.07552v1. SPARE also cuts recurrent-kernel DRAM traffic 7.49x and peak GPU memory 3.31x, while preserving better associative-retrieval accuracy than structured pruning at matched memory budgets. The method targets the quadratic scaling of recurrent states with representation dimension and the repeated data transfers that create a GPU memory bottleneck during decoding, using heterogeneous rank allocation and lightweight write compensation without online decomposition or dense-state reconstruction.", "body_md": "arXiv:2611.07552v1 Announce Type: new \nAbstract: Linear attention reduces the quadratic sequence-length complexity of softmax attention by maintaining fixed-size recurrent states. However, these states scale quadratically with the representation dimension and require repeated data transfers during decoding, creating a substantial GPU memory bottleneck. Recent studies have revealed significant low-rank redundancy in recurrent states, suggesting opportunities for more efficient representations. We introduce SPARE (State Projection for Accelerated Recurrent Execution), a training-free method that exploits this redundancy through fixed low-dimensional subspaces, heterogeneous rank allocation, and lightweight write compensation. By performing recurrent reads and updates directly on compact states, SPARE reduces memory traffic without online decomposition or dense-state reconstruction. SPARE achieves 2.55x full-model decoding speedup, 7.49x lower recurrent-kernel DRAM traffic, and 3.31x lower peak GPU memory, while preserving better associative-retrieval accuracy than structured pruning at matched memory budgets.", "url": "https://wpnews.pro/news/spare-fast-linear-attention-decoding-with-compact-recurrent-states", "canonical_source": "https://www.machinebrief.com/news/spare-fast-linear-attention-decoding-with-compact-recurrent-g8z0", "published_at": "2026-11-06 05:00:00+00:00", "updated_at": "2026-11-06 05:21:04.194335+00:00", "lang": "en", "topics": ["machine-learning", "ai-research", "ai-infrastructure", "large-language-models"], "entities": ["SPARE", "State Projection for Accelerated Recurrent Execution", "arXiv"], "also_reported_by": [], "alternates": {"html": "https://wpnews.pro/news/spare-fast-linear-attention-decoding-with-compact-recurrent-states", "markdown": "https://wpnews.pro/news/spare-fast-linear-attention-decoding-with-compact-recurrent-states.md", "text": "https://wpnews.pro/news/spare-fast-linear-attention-decoding-with-compact-recurrent-states.txt", "jsonld": "https://wpnews.pro/news/spare-fast-linear-attention-decoding-with-compact-recurrent-states.jsonld"}}