Accelerating Attention for Ultra-Long Context

A new attention architecture called Latent Sparse Attention (LSA) matches full-attention accuracy at 10M tokens while cutting attention-layer prefill time by 72.4× on an H200 GPU when distilled into Qwen-3.8-27B, according to the research team behind it. LSA clusters keys hierarchically by similarity so each query scores only a few clusters, making attention subquadratic; at 10M tokens it finds which tokens to attend to 22.7× faster than existing sparse attention, which spends 98% of its time on that search. End-to-end serving in SGLang on 8× H200 GPUs improves time to first token by 9.2× at 10M tokens, and both distilled models stay within two points of the originals on knowledge and reasoning benchmarks.

Accelerating Attention for Ultra-Long Context Subquadratic attention that keeps full-attention accuracy to 10M tokens The attention mechanism is responsible for most of the progress in AI over the last decade. However, it becomes prohibitively costly as model context grows, which blocks reasoning at ultra-long context. We overcome this problem with Latent Sparse Attention LSA , a step towards subquadratic attention that matches accuracy of full attention at ultra-long contexts 10M tokens . LSA can be added to existing models with a short distillation, or trained from scratch in new models. Agentic workflows are increasingly shifting towards 1M+ tokens , such as open-ended scientific research and managing code repositories. However, a major bottleneck to progress is attention, as its cost scales quadratically with context. This is because in attention, every query token attends to every preceding token, which means 1M+ operations per new token added. This can be tackled at two levels. At the level of the GPU kernel, methods such as , skip work that would barely change the result. At the level of the architecture, methods let each query look at only a few of the most relevant tokens, but they either don't perform well at long context, or their costs still scale quadratically. In fact, at 10M tokens, 98% of sparse attention's time is spent finding which tokens to attend to. LSA is an architectural approach, and does this 22.7× faster, while using slightly less memory. We run our experiments on both H200 and B300 GPUs, one layer at a time and end to end. How LSA works LSA selects the most relevant keys for a query efficiently, without checking every preceding key. It does this because it structures the keys in a hierarchy of clusters, so that a single key can then be retrieved hierarchically. Critically, keys are clustered according to how similar they are to each other, which means similarly relevant keys get selected for attention. At each level the query scores only a few clusters, and attention is computed over a fixed number of keys regardless of context size. This is what makes ultra-long context practical. When the clustering is fully hierarchical, attention goes from quadratic in cost to subquadratic . Two variants: D-LSA and N-LSA - D-LSA - Distilled LSA replaces attention in a pre-trained model with LSA, through a short . - N-LSA - Native LSA puts LSA in a model from the start and trains it from scratch. Results D-LSA We measure speed at 10M tokens on both H200 and B300 GPUs, first for a single attention layer, then for the whole model served on eight GPUs. When D-LSA is added to Qwen-3.8-27B, it speeds up the prefill of its attention layers by 72.4× on H200 9.2× in end-to-end serving . When it replaces the existing sparse attention in GLM-5.3-Flash, it speeds up the prefill of its attention layers by 22.7× on H200 2.2× in end-to-end serving . On knowledge and reasoning benchmarks, both models stay within two points of the original. On long-context tasks, Qwen keeps pace on RULER up to 10M tokens, and its 5-point drop on LongBench v2 is within the 95% confidence interval. GLM trails on RULER by up to about 8 points at 2M–5M tokens, and is within 4 at 10M, against a baseline that was also fine-tuned. Attention layer speedup over Full Attention One layer on one H200 or B300 | | H200 | | B300 | | |---|---|---|---|---| | | 1M | 10M | 1M | 10M | | Prefill | 8.2× | 72.4× | 6.1× | 42.9× | | Decode | 5.2× | 42.5× | 2.8× | 18.4× | End-to-end speedup over Full Attention Whole model in SGLang on 8× H200 or B300 | | H200 | | B300 | | |---|---|---|---|---| | | 1M | 10M | 1M | 10M | | Time to first token | 1.5× | 9.2× | 1.0× | 5.7× | | Time per output token | 1.0× | 2.9× | 0.7× | 1.5× | Missing optimised kernels for baseline model on Blackwell under parallelism Setup Attention layer - GPU - One H200 / B300, bf16 - Prefill - Last 32k chunk - Decode - One token - Baseline - FA3; TRT-LLM on B300 End to end - GPUs - 8× H200 / B300, bf16 - Mesh - TP8 + DCP8 on H200 - Serving - SGLang, 1 request - Baseline - Full Attention Accuracy on RULER % - average over all 13 tasks Long-context synthetic tasks by NVIDIA BaselineD-LSA Accuracy on downstream tasks % | | Baseline | D-LSA | Δ | |---|---|---|---| | MMLU-Pro | 83.2 | 83.0 | −0.2 | | GPQA-D | 88.4 | 87.9 | −0.5 | | AIME25 | 98.3 | 97.5 | −0.8 | | MATH500 | 98.0 | 99.0 | +1.0 | | LongBench v2 | 70.0 | 65.0 | −5.0 | Attention layer speedup over GLM's Sparse Attention One layer on one H200 or B300 | | H200 | | B300 | | |---|---|---|---|---| | | 1M | 10M | 1M | 10M | | Prefill | 3.5× | 22.7× | 2.8× | 12.5× | | Decode | 1.3× | 4.6× | 1.2× | 5.2× | End-to-end speedup over GLM's Sparse Attention Whole model in SGLang on 8× H200 or B300 | | H200 | | B300 | | |---|---|---|---|---| | | 1M | 10M | 1M | 10M | | Time to first token | 1.0× | 2.2× | 1.0× | 1.6× | | Time per output token | 1.0× | 2.1× | 1.0× | 2.2× | Setup Attention layer - GPU - One H200 / B300, bf16 - Prefill - Last 32k chunk - Decode - One token - Baseline - GLM's DSA indexer End to end - GPUs - 8× H200 / B300, bf16 - Mesh - CP8 + EP8 - Serving - SGLang, 1 request - Baseline - GLM's DSA indexer Accuracy on RULER % - average over all 13 tasks Long-context synthetic tasks by NVIDIA BaselineD-LSA Accuracy on downstream tasks % | | Baseline | D-LSA | Δ | |---|---|---|---| | MMLU-Pro | 85.4 | 85.7 | +0.3 | | GPQA-D | 85.9 | 86.9 | +1.0 | | AIME25 | 97.5 | 95.8 | −1.7 | | MATH500 | 98.0 | 98.0 | 0.0 | | LongBench v2 | 70.0 | 68.3 | −1.7 | N-LSA When training from scratch with N-LSA layers, in our tests at up to 128k tokens, we match the accuracy of Full Attention but train faster. Compared to the other sparse attention methods we tested, N-LSA is the only one that matches Full Attention's accuracy. Validation loss Lower is better Full Attention 1.085 SWA 1.153 NSA 1.117 PISA 1.100 N-LSA 1.085 All methods start within 0.004 loss of each other at 0 h. Natively Trained A hybrid model of about 0.5B parameters, trained from scratch on next-token prediction alone with a 100B token warmup at 4k context, then 12B tokens at 128k. Accuracy on RULER % - average over all 13 tasks Long-context synthetic tasks by NVIDIA Accuracy on downstream tasks % | | Commonsense | | | Recall | | |---|---|---|---|---|---| | | ARC-e | ARC-c | HellaSwag | SWDE | FDA | | Full Attention | 50.0 | 30.0 | 45.6 | 67.8 | 79.3 | | SWA | 53.9 | 31.4 | 45.2 | 64.6 | 69.6 | | NSA | 53.8 | 30.1 | 45.8 | 62.2 | 72.2 | | PISA | 52.2 | 28.7 | 45.5 | 63.7 | 78.6 | | N-LSA | 54.6 | 30.8 | 45.5 | 66.4 | 79.2 | PaperTrail: towards more realistic long-context tasks To evaluate models on long-context science tasks, we present a new eval called PaperTrail. It consists of a trail of related synthetic facts hidden in a large collection of recent scientific papers. Correct answers require finding the facts and connecting them, like a literature search. We show that LSA improves Qwen-3.8-27B science fact-finding at long context. On GLM-5.3-Flash it trails the original by about 6 to 7 points from 2M to 10M tokens. Summary LSA provides a practical way forward for ultra-long context attention, both for distilling pre-trained models and for native training. We integrate LSA into SGLang with context parallelism, provide GPU kernels for H200 and B300, and release the code with all training and inference details. Please see our paper for full details on the method, as well as a variety of other cool results of LSA. Citation @misc{kubicek2026lsa, title = {Latent Sparse Attention: Towards Subquadratic Ultra-Long Context Attention}, author = {Ales Kubicek and Andrea Caciolai and Liangzu Peng and Sukrit Sumant and Benjamin Hoover and Fraser Greenlee and Ruairidh M. Battleday and Radim Urban and Torsten Hoefler and James C. R. Whittington}, year = {2026}, eprint = {2610.42147}, archivePrefix = {arXiv}, primaryClass = {cs.AI}, url = {https://arxiv.org/abs/2610.42147} }