Joint Subspace Sparse Attention for Efficient Long-Context Decoding

Joint Subspace Sparse Attention (JSSA), a new token-level sparse attention method, reaches up to 6.43x the decoding throughput of full-KV attention at 128K context on an NVIDIA B200 GPU, according to the arXiv paper 2610.19026v1. JSSA learns a shared low-dimensional query-key subspace by minimizing token-score preservation error and uses a gather-free token-level indexed-attention kernel supporting distinct per-KV-head selections under the standard grouped-query attention KV-cache layout. The method stays close to full-KV accuracy across standard GQA, hybrid sliding-window/global attention, and Multi-head Latent Attention models, and reaches 96-99% of model-native DSA decoding throughput at 700B scale without model training.

arXiv:2610.19026v1 Announce Type: new Abstract: Sparse attention reduces the KV-cache traffic that dominates long-context LLM decoding, but practical methods largely operate at coarse granularity. Token-level selection offers finer control, yet presents two challenges: low-dimensional scoring may lose query--key structure needed for accurate ranking, while gathering scattered selected tokens can offset the resulting latency benefit. We introduce Joint Subspace Sparse Attention JSSA to address both challenges. For selection quality, JSSA learns a shared low-dimensional query--key subspace by minimizing token-score preservation error. We upper-bound the resulting non-convex objective with a tractable marginal objective that admits a closed-form spectral solution; direct refinement of the exact objective provides no measurable improvement in the tested settings. For execution efficiency, we develop a gather-free token-level indexed-attention kernel that supports distinct per-KV-head selections under the standard grouped-query attention GQA KV-cache layout. Across standard GQA, hybrid sliding-window/global attention, and Multi-head Latent Attention MLA models, JSSA consistently remains close to full-KV accuracy while avoiding the substantial generation-length inflation and architecture-dependent failures observed with competing methods. On an NVIDIA B200 GPU, JSSA achieves up to $6.43\times$ the decoding throughput of full-KV attention at 128K context. JSSA achieves reasoning accuracy comparable to model-native DSA without model training and reaches 96--99\% of its decoding throughput even at 700B scale large models.