NVIDIA published an eight-guideline sparse attention co-design guide on November 17, 2026 aimed at model developers training sparsity into long-context transformers. The post is not a product launch; it explains where time goes at 128K tokens and how indexer knobs interact with Tensor Cores, HBM bandwidth, and speculative decode depth.
If you ship coding agents or research copilots with 100K+ system prompts, the headline is blunt: sparse core attention does not erase long-context cost — it moves FLOPs into the indexer and Top-k. explainx.ai covered related indexer debates in Latent Sparse Attention speedup claims, SubQ at 12M context, and SageMaker KV-aware routing.
What problem sparse attention solves — and what it does not #
Dense attention scores every query against every key, so per-layer prefill cost grows with context length N. At 128K tokens, NVIDIA’s authors say attention can consume most of a layer’s prefill time.
Sparse attention lets each query attend to only k keys, chosen per query, so core attention scales with k, not N. The catch: something must find those keys. NVIDIA calls that block the indexer. It still scores all keys (or compressed keys), then Top-k keeps the winners, then core attention runs softmax on the subset.
Per-layer prefill time at 128K tokens comparing dense and sparse attention on NVIDIA GPUs
Image: NVIDIA NVIDIA’s Figure 1 compares DeepSeek-R1 dense MLA with DeepSeek V3.2 DSA at ISL = 128K. In the sparse case, indexer + Top-k take 47% of per-layer prefill while core attention falls to 24%. Builders who only optimize FlashAttention-style kernels for the core stage can miss half the layer.
Three stages you must budget together #
Three stages of sparse attention indexer Top-k and core attention for long context inference
Image: NVIDIA NVIDIA breaks production sparse attention into:
| Stage | What it does | Cost driver at long context |
|---|---|---|
| Indexer | Scores keys (often FP8 matmul + fused epilogue) | Still linear in keys scored |
| Top-k | Selects highest logits per query | Grows with keys scored, weakly with k |
| Core attention | Softmax over chosen keys/values | Scales with selected set size |
The post contrasts model-level sparsity (trained in) with bolt-on methods such as MInference or Quest. It focuses on the trained path because indexer architecture and GPU kernels must match.
Recent models in NVIDIA’s Table 1 include DeepSeek DSA, GLM 5.3 Flash, NSA, Qwen QSA, and MiniMax MSA — each with different query heads, compression ratio (CR), and token- vs block-level selection. For GLM-scale context work, see our GLM 5.5 lecture notes; for Meituan sparse stacks, see LongCat sparse attention.
Indexer arithmetic intensity: prefill vs decode #
Normalized indexer runtime versus group size G on NVIDIA GPUs
Image: NVIDIA NVIDIA models the indexer as one batched matmul with a fused epilogue (ReLU, cross-head reduction, logit store). Two lessons drive the guidelines:
- Decode is memory-bound — reading thekey cache dominates; extra query heads add FLOPs butlittle new traffic , so decode runtime can stay flat asgroup size G grows until you exhaustTensor Core tile rows .
- Prefill is compute-bound at moderate G — butFP32 logit writes cap gains at small G until the matmul dominates.
Speculative decoding scores D+1 query rows per step against the same cache. NVIDIA reports near-free draft rows while G × (D+1) ≤ ~200 (FP32 epilogue) or 256 (FP16 epilogue). Beyond that, kernels launch extra CTAs and runtime steps up.
Guideline 1–2: Tile fit and FP16 epilogue
Guideline 1: keep G × (D+1) inside one Tensor Core tile (~200 rows, ~256 with FP16 epilogue).
Guideline 2: run the indexer epilogue in FP16 (and FP16 accumulate when QK scaling keeps matmul outputs in range). Measured decode speedups reach 1.31× (FP16 epilogue) and 1.45× (FP16 accumulate) at G = 64, D+1 = 4.
Guideline 3–4: KH and head dimension
Guideline 3: prefer KH = 1 with cross-head reduction so you write one logit per key. Raising indexer key heads makes decode cost grow linearly (NVIDIA cites 7.6× runtime from KH 1→8) because each head reads the full cache.
Guideline 4: use Hsz = 128 or 256. Hsz = 64 leaves Tensor Cores waiting on the epilogue in prefill; decode cost tracks extra key bytes as Hsz grows.
Guideline 5–6: Fewer keys scored; fuse Top-k
Guideline 5: cut keys the indexer scores via compression (CR), hierarchical indexing, or index sharing across layers — methods named in Table 1 for GLM and LongCat variants.
Guideline 6: treat Top-k as part of the indexer budget. Its cost tracks keys scored, not final k. NVIDIA cites 2.1×–2.4× decode Top-k wins from Guess-Verify-Refine Top-k in TensorRT LLM when reusing the prior step’s selection, and FlashInfer paths that never materialize full logit matrices.
For serving stacks, pair these ideas with NVIDIA Dynamo on Kubernetes and vLLM disaggregated KV on AWS.
Guideline 7–8: Block selection and head sharing
Token-level selection scatters KV reads; block-level selection lets core attention fetch contiguous blocks once and share them across queries. NVIDIA warns halving block size while holding 2,048 selected keys can nearly double core-attention runtime because more blocks mean more metadata and partial merges.
Guideline 8: use a large core-attention group size so many query heads share each KV head and reuse block fetches.
What model developers should do this week #
- Profile at your real ISL — if indexer+Top-k exceed core attention near128K , kernel work on dense flash alone will not fix latency.
- Align indexer G and speculative depth withTensor Core tile limits before chasing wider epilogues.
- Prototype Top-k fusion usingTensorRT LLM orFlashInfer references in the NVIDIA post before writing custom CUDA.
- Log block size vs token-level selection when portingNSA/QSA-style models to vLLM — block tuning changesboth indexer keys and core fetch patterns.
- Pair model sparsity with routing — prefix caches and KV-aware routers (SageMaker post ) attackrepeated prefill , notper-layer indexer FLOPs .
How this fits the November inference stack #
Sparse attention is half of the long-context story; the other half is orchestration. The same week, AWS published managed NVIDIA Dynamo on SageMaker HyperPod Inference and AgentCore JSON optimizations for megabyte tool payloads — orchestration layers that sit above indexer design. NVIDIA’s guidelines help model teams; Dynamo and AgentCore patterns help platform teams.
If you run Rubin-era targets, also read NVIDIA Rubin HPC specs and vLLM Rubin throughput claims — hardware bandwidth still caps indexer decode even with perfect sparsity math.
Related reading #
Worked example: reading NVIDIA Figure 4 on group size #
NVIDIA normalizes indexer runtime against group size G (query heads per key head) while holding other knobs fixed. In prefill, runtime climbs with G because more query rows participate in the batched matmul. In decode, runtime barely moves when G grows from small values to 16× larger groups, because the single key-cache read sets the wall clock.
That asymmetry is why Guideline 1 targets decode tile packing: teams running speculative decoding should count G × (D+1) rows before adding draft depth, not after latency regresses. If you profile with Nsight or DCGM and see decode bound on HBM %, shrinking KH matters more than shaving core attention softmax kernels.
For agent traces with bursty tool returns, prefill-bound turns still happen when a new 64K JSON blob lands. Sparse core attention helps, but indexer epilogue and logit writes can dominate at moderate G — exactly the 2% vs 16× headroom curve in the post’s Figure 4 discussion.
Comparison table: indexer choices in the wild #
| Method (NVIDIA table) | Selection | Notable knob |
|---|---|---|
| DeepSeek DSA | Token | Separate FP8 index cache |
| GLM 5.3 Flash | Block of 4 + tail | CR = 4 |
| NSA | Block of 16 | 64 query heads |
| MiniMax MSA | Block of 128 | MaxPool, no cross-head reduction |
| MoBA | Block of 4096 | Pools keys before scoring |
When you port a checkpoint trained with one row of this table, changing CR or block size without retraining usually fails quality gates — co-design is about training-time alignment with kernels you will ship, not a post-hoc toggle.
FAQ for platform leads #
Should we block sparse attention on older GPUs? NVIDIA’s sweeps use FP8 indexer kernels on recent datacenter cards. Validate on your SKU before promising 128K in marketing; llama.cpp sparse attention notes show community ports lag vendor guides.
Does sparse attention replace quantization? No. NVFP4 and weight-only schemes like NVIDIA Qwen3.8 Flash quantized builds cut memory; sparsity cuts attention FLOPs on long keys. Production stacks combine both.
Where do I test without training a model? Start with open sparse checkpoints (DeepSeek-class, GLM-class) on vLLM or TensorRT-LLM, enable long prompts, and compare prefill breakdown with and without sparsity enabled — then read NVIDIA’s Guideline 6 on whether Top-k is fused in your build.
Sources: NVIDIA technical blog, November 17, 2026 · TensorRT LLM repository · FlashInfer