{"slug": "adamas-hadamard-sparse-attention-for-efficient-long-context-inference", "title": "Adamas: Hadamard Sparse Attention for Efficient Long-Context Inference", "summary": "Researchers introduced Adamas, a token-level sparse attention mechanism that applies the Hadamard transform, bucketization and 2-bit compression to select which key-value tokens to attend to during long-context inference, per arXiv paper 2510.18413v2. Adamas delivers up to 6.5x self-attention speedup and 1.7x end-to-end decoding speedup at a 32K context length while maintaining near-lossless performance within the effective context length of Llama-3.1-8B-Instruct, the authors report. The method also holds performance comparable to full attention on long-form reasoning benchmarks including AIME 2024 and AIME 2025.", "body_md": "arXiv:2510.18413v2 Announce Type: replace \nAbstract: Large language models (LLMs) now support context windows of hundreds of thousands to millions of tokens, enabling applications such as long-document summarization, large-scale code synthesis, multi-document question answering and persistent multi-turn dialogue. However, such extended contexts exacerbate the quadratic cost of self-attention, leading to severe latency in autoregressive decoding. Existing sparse attention methods reduce these costs by attending to only a subset of key-value (KV) tokens, but fixed or coarse-grained selection strategies can miss important tokens or retain substantial redundancy under constrained token budgets. We introduce \\textbf{Adamas}, an accurate and efficient token-level sparse attention mechanism for long-context inference. Adamas applies the Hadamard transform, bucketization and 2-bit compression to produce compact representations, and uses lightweight distance-based estimation for efficient top-$k$ candidate selection. The selected positions are then used to gather the corresponding original KV states for sparse attention. Comprehensive evaluations show that Adamas maintains near-lossless performance within the effective context length of Llama-3.1-8B-Instruct. In efficiency evaluations, Adamas achieves up to $6.5\\times$ self-attention speedup and $1.7\\times$ end-to-end decoding speedup at a 32K context length. Adamas also maintains performance comparable to full attention on challenging long-form reasoning benchmarks, including AIME 2024 and AIME 2025.", "url": "https://wpnews.pro/news/adamas-hadamard-sparse-attention-for-efficient-long-context-inference", "canonical_source": "https://www.machinebrief.com/news/adamas-hadamard-sparse-attention-for-efficient-long-context-qwqw", "published_at": "2026-11-09 05:00:00+00:00", "updated_at": "2026-11-09 05:23:02.195023+00:00", "lang": "en", "topics": ["artificial-intelligence", "machine-learning", "large-language-models", "ai-research", "ai-infrastructure"], "entities": ["Adamas", "Llama-3.1-8B-Instruct", "arXiv", "AIME 2024", "AIME 2025"], "also_reported_by": [], "alternates": {"html": "https://wpnews.pro/news/adamas-hadamard-sparse-attention-for-efficient-long-context-inference", "markdown": "https://wpnews.pro/news/adamas-hadamard-sparse-attention-for-efficient-long-context-inference.md", "text": "https://wpnews.pro/news/adamas-hadamard-sparse-attention-for-efficient-long-context-inference.txt", "jsonld": "https://wpnews.pro/news/adamas-hadamard-sparse-attention-for-efficient-long-context-inference.jsonld"}}