ADIndex: Layer-Adaptive Sparse Attention Indexing via Group-Level Distillation

A new arXiv paper (2611.12956v1) proposes ADIndex, a layer-adaptive sparse attention indexing method that cuts the indexer's share of long-context prefill latency, which the authors report can exceed 50% of total prefill GPU time. Tested on DeepSeek-V3.2 and GLM-5.1, ADIndex preserved near-lossless performance on LongBench, LongBench v2 and RULER while delivering prefill and decoding kernel speedups of up to 6.22x and 3.40x at a 256K context length, plus a 1.54x end-to-end serving speedup at 128K. The method combines layer-wise calibration, group-level distillation and token bank sharing within query groups to avoid repeated full-prefix scans.

arXiv:2611.12956v1 Announce Type: new Abstract: DeepSeek Sparse Attention DSA reduces the computational cost of attention through token-level selection. However, its indexer still scores the entire causal prefix for each query, making indexing a dominant source of latency in long-context inference. Notably, the indexer alone can account for more than 50% of total prefill GPU time at long context lengths. To address this bottleneck, we propose ADIndex, an Adaptive and Distillation-driven Indexing method for token-level sparse attention. ADIndex integrates three key components: i a layer-wise calibration strategy that adaptively accommodates cross-query retrieval overlap, motivated by our observation of pronounced inter-layer heterogeneity; ii a group-level distillation scheme that leverages this heterogeneity to efficiently train adaptive layer-wise group indexers, capturing salient retrieval signals from individual queries; and iii a low-latency inference strategy that reduces repeated full-prefix scans through token bank sharing within query groups while retaining query-specific refinement. Experiments on DeepSeek-V3.2 and GLM-5.1 demonstrate that ADIndex preserves near-lossless task performance on LongBench, LongBench v2, and RULER while delivering substantial efficiency gains over standard DSA, including prefill and decoding kernel speedups of up to 6.22x and 3.40x, respectively, at a context length of 256K across the layer-wise reuse configurations, as well as an end-to-end serving speedup of 1.54x at a context length of 128K.