{"slug": "lisa-linear-indexed-sparse-attention-for-efficient-long-context-reasoning", "title": "LISA: Linear-Indexed Sparse Attention for Efficient Long-Context Reasoning", "summary": "Researchers propose LISA (Linear-Indexed Sparse Attention), a plug-and-play attention module that reduces inference complexity from O(n^2) to O(nM) for long-context reasoning models. Experiments on DeepSeek-distilled-Qwen models show LISA achieves a 50% inference speedup under 16K-token context while improving average performance by 5.6% on reasoning benchmarks including AIME and MATH-500.", "body_md": "arXiv:2607.19358v1 Announce Type: new\nAbstract: Recent advances in long chain-of-thought reasoning models such as DeepSeek-R1 have led to increasingly longer inference context lengths under the test-time scaling paradigm. However, the O(n^2) computational complexity of standard self-attention causes inference costs to grow sharply with long sequences, limiting the deployment of long-CoT reasoning in production settings. To address this, we propose LISA (Linear-Indexed Sparse Attention), a plug-and-play attention replacement module that requires no pretraining from scratch. LISA integrates two lightweight components in parallel within the original model: (1) a Linear Attention module that provides long-range memory with O(n) time complexity; (2) a Lightning Indexer that selects the top-M important tokens from the full context to feed into a Sparse Self-Attention. The two branches are fused via a gating mechanism, reducing inference complexity from O(n^2) to O(nM) (M << n) for generating n tokens. We design a two-stage training pipeline: Stage 1 initializes the model by integrating the linear attention to capture long-range dependencies, complemented by a sliding-window attention mechanism that is optimized via knowledge distillation to approximate the full self-attention distribution of a frozen teacher model. In Stage 2, we further introduce the Indexer to replace the static sliding-window mechanism, enabling dynamic token selection from broader contexts. The Indexer is trained using a novel per-head KL divergence loss, which aligns its selection behavior with the attention patterns of the teacher model. Experiments on DeepSeek-distilled-Qwen models demonstrate that LISA achieves a 50% inference speedup under 16K-token context, while improving average performance by 5.6% on reasoning benchmarks including AIME and MATH-500.", "url": "https://wpnews.pro/news/lisa-linear-indexed-sparse-attention-for-efficient-long-context-reasoning", "canonical_source": "https://arxiv.org/abs/2607.19358", "published_at": "2026-07-23 04:00:00+00:00", "updated_at": "2026-07-23 04:08:33.286042+00:00", "lang": "en", "topics": ["artificial-intelligence", "machine-learning", "large-language-models", "ai-research"], "entities": ["LISA", "DeepSeek", "Qwen", "AIME", "MATH-500"], "alternates": {"html": "https://wpnews.pro/news/lisa-linear-indexed-sparse-attention-for-efficient-long-context-reasoning", "markdown": "https://wpnews.pro/news/lisa-linear-indexed-sparse-attention-for-efficient-long-context-reasoning.md", "text": "https://wpnews.pro/news/lisa-linear-indexed-sparse-attention-for-efficient-long-context-reasoning.txt", "jsonld": "https://wpnews.pro/news/lisa-linear-indexed-sparse-attention-for-efficient-long-context-reasoning.jsonld"}}