Beyond GQA: Characterizing Modern Attention Variants with ServingStudio A ServingStudio analysis of eleven recent open models found that at a 128K-token context, newer attention designs need up to 139× less attention state per request, load up to 250× fewer bytes per decode step, and require up to 183× fewer prefill attention FLOPs than Qwen3-Coder-480B-A35B, a grouped-query attention (GQA) baseline of similar size. DeepSeek V4.1 Flash delivered the largest memory and decode gains, with attention running up to 44× faster in decode and up to 46× faster in prefill, while Qwen3.8 Flash Next posted the 183× prefill FLOP reduction. The authors conclude that bottlenecks and optimization opportunities identified in GQA models do not always carry over, and that inference research should study the new architectures — latent caches, sliding windows, sparse attention, compressed caches and linear attention — directly. All posts https://servingstudio.cs.washington.edu/blog.html Beyond GQA: Characterizing Modern Attention Variants with ServingStudio Top: the eleven attention designs in today's open models. Bottom: the best new model on each measure, against the GQA baseline at 128K tokens. Introduction Much of today's performance analysis of LLM inference still rests on models with grouped-query attention GQA , in which multiple Query vectors attend to the same Key-Value vector to save memory. That analysis tells us when a step is bound by attention and when by the FFN, how many requests fit in memory, and which kernels are worth tuning. Today's open models, however, have moved beyond GQA. They combine latent caches, sliding windows, sparse attention, compressed caches and linear attention. These designs drastically change the HBM consumption, the memory loading, and the TFLOPs spent on the attention. In this post, we look at eleven recent models: DeepSeek V4.1 Flash, DeepSeek V4 Pro, GLM-5.3-Flash, Qwen3.8 Flash Next, MiMo V2.6 Flash, Inkling, GLM-5.3, MiniMax-M3.1 Flash, Kimi K3, Kimi K3.1 Pro and Qwen3.8 27B. We compare them with Qwen3-Coder-480B-A35B, a GQA model of similar size to most of them. At a 128K-token context, the new models need far less than the baseline: - Memory. A request keeps up to 139× less attention state DeepSeek V4.1 Flash . - Decode. Each step loads up to 250× fewer bytes, and its attention runs up to 44× faster DeepSeek V4.1 Flash . - Prefill. A prompt needs up to 183× fewer attention FLOPs Qwen3.8 Flash Next , and its attention runs up to 46× faster DeepSeek V4.1 Flash . These changes matter beyond the numbers themselves: they change the answers that the analysis of GQA models gives. Our main takeaway The attention landscape has changed, and inference research needs to catch up. New attention designs change how much memory a request needs, how many bytes each step moves, and how much it computes. As a result, the bottlenecks and optimization opportunities of GQA models do not always carry over. Future inference research should study these new models directly, instead of reusing conclusions drawn from GQA models. Architectures All models except Qwen3.8 27B are mixture-of-experts MoE transformers. Their attention layers differ. The baseline uses GQA in every layer. GLM-5.3 replaces GQA with DSA in every layer, and MiniMax-M3.1 Flash replaces it with MSA. The other nine models mix two or more kinds of attention layers. The table below lists their size and attention choices. Figure data as text: The models and their attention layers Models | Model | Role | Facts | Attention layers | |---|---|---|---| | Qwen3-Coder-480B-A35B | Baseline | 480B total; 35B active; 262K context | Grouped-query attention GQA ×62 Full attention | | MiMo V2.6 Flash | — | 309B total; 15B active; 1M context | Sliding-window attention SWA ×39 Sliding window ; Grouped-query attention GQA ×9 Full attention | | Inkling | — | 975B total; 41B active; 1M context | Sliding-window attention SWA ×55 Sliding window ; Grouped-query attention GQA ×11 Full attention | | GLM-5.3 | — | 753B total; ~40B active; 1M context | DeepSeek Sparse Attention DSA ×78 Sparse attention | | MiniMax-M3.1 Flash | — | ~428B total; ~23B active; 1M context | MiniMax Sparse Attention MSA ×60 Sparse attention | | DeepSeek V4 Pro | — | 1.6T total; 49B active; 1M context | Compressed Sparse Attention CSA ×30 Sparse attention ; Heavily Compressed Attention HCA ×31 Full attention | | DeepSeek V4.1 Flash | — | 552B backbone; 8B active in prefill; 16B active in decode; 1M context | Sliding-window attention SWA ×40 Sliding window ; Compressed Sparse Attention CSA2 ×38 Sparse attention | | Kimi K3 | — | 2.8T total; 104B active; 1M context | Kimi Delta Attention KDA ×69 Linear attention ; Gated MLA ×24 Full attention | | Kimi K3.1 Pro | — | 2.8T total; 104B active; 1M context | Kimi Delta Attention KDA ×69 Linear attention ; Gated DSA ×24 Sparse attention | | GLM-5.3-Flash | — | 320B total; 18B active; 1M context | Kimi Delta Attention KDA ×34 Linear attention ; DeepSeek Sparse Attention DSA ×11 Sparse attention | | Qwen3.8 Flash Next | — | 125B total; 6B active; 51B n-gram embedding; 262K context, 1M with YaRN | Gated DeltaNet GDN ×36 Linear attention ; Qwen Sparse Attention QSA ×12 Sparse attention | | Qwen3.8 27B | — | 27B dense; 262K context, 1M with YaRN | Gated DeltaNet GDN ×48 Linear attention ; Gated attention GQA ×16 Full attention | All of this figure's data every setting and operation : data/model cards.md https://servingstudio.cs.washington.edu/blog/attention-variants/assets/interactive/data/model cards.md All analysis below assumes the model is deployed on B200 GPUs with data-parallel DP attention: each GPU runs every attention head of a layer, and each request stays on one GPU. We borrow kernels from vLLM or SGLang and measure them with ServingStudio. Attention Variants We first go through how each attention design works. They fall into two branches. The first keeps an entry for every earlier token in the cache. On this branch, GQA and MLA make each entry smaller, while sliding windows, DSA, CSA and HCA cut how much of the cache is kept or read. The second branch, linear attention, folds the whole context into a state of fixed size. Grouped-query attention GQA Figure data as text: GQA in one layer of Qwen3-Coder-480B-A35B Prefill: data flow - Main row: x N × 6,144 → Projection → QK-norm → RoPE → Scores → Causal mask → Softmax → Weighted sum → Output projection → out to the MoE FFN - Write: RoPE k and Projection v no norm, no RoPE → Write K, V → KV cache, kept for the whole request - Scores reads K from the cache: reads K: L earlier + N in chunk - Weighted sum reads V from the cache: reads V: L earlier + N in chunk Decode: data flow - Main row: x B × 6,144 → Projection → QK-norm → RoPE → Scores → Softmax → Weighted sum → Output projection → out to the MoE FFN - Write: RoPE k and Projection v no norm, no RoPE → Write K, V → KV cache, kept for the whole request - Scores reads K from the cache: reads K: all L tokens - Weighted sum reads V from the cache: reads V: all L tokens Cache - Prefill: KV cache, kept for the whole request; rows: K, V - Decode: KV cache, kept for the whole request; rows: K, V - 8 KV heads × 128 × K + V = 2 KiB per token and layer in FP8; 124 KiB per token over 62 layers All of this figure's data every setting and operation : data/flow gqa.md https://servingstudio.cs.washington.edu/blog/attention-variants/assets/interactive/data/flow gqa.md In standard multi-head attention, every query head has its own key head and value head. GQA lets a group of query heads share one key-value head, so each token keeps fewer keys and values. Our baseline, Qwen3-Coder-480B-A35B, has 96 query heads and 8 key-value heads in each layer, so 12 query heads share each key-value head. The token's hidden state is first projected into queries, keys and values, and QK-norm and RoPE normalize the queries and keys and encode their positions. The keys and values are stored in the KV cache, where they stay for the entire lifecycle of the request. To compute its output, a new token compares its queries with the keys of all earlier tokens, turns these scores into weights with a softmax, and takes the weighted sum of the earlier tokens' values. The two blue arrows show these two reads from the cache: computing the scores reads every key, and the weighted sum reads every value. Multi-head latent attention MLA Figure data as text: Gated MLA in one layer of Kimi K3 Prefill: data flow - Main row: x N × 7,168 → Down-projection → RMSNorm → Query up-projection → Scores → Causal mask → Softmax → Weighted sum → Output gate → Output projection → out to the MoE FFN - Parallel branch: from x x → Gate projection → joins Output gate g - Parallel branch: from Query up-projection → Expand K, V → joins Scores K - V goes from Expand K, V straight to Weighted sum - Write: RMSNorm c kv and Down-projection k s, not normalized → Write c kv, k s → Latent cache, kept for the whole request - Expand K, V reads c kv, k s from the cache: reads c kv, k s: L earlier + N in chunk Decode: data flow - Main row: x B × 7,168 → Down-projection → RMSNorm → Query up-projection → Absorb W UK → Scores → Softmax → Weighted sum → Absorb W UV → Output gate → Output projection → out to the MoE FFN - Parallel branch: from x x → Gate projection → joins Output gate g - q s goes from Query up-projection straight to Scores - Write: RMSNorm c kv and Down-projection k s, not normalized → Write c kv, k s → Latent cache, kept for the whole request - Scores reads c kv, k s from the cache: reads c kv, k s: all L tokens - Weighted sum reads c kv, k s from the cache: same c kv, as the values Cache - Prefill: Latent cache, kept for the whole request; rows: c kv, k s - Decode: Latent cache, kept for the whole request; rows: c kv, k s - c kv 512 numbers and k s 64 , read by all heads: 576 B per token and layer in FP8; 13.5 KiB over 24 layers All of this figure's data every setting and operation : data/flow mla.md https://servingstudio.cs.washington.edu/blog/attention-variants/assets/interactive/data/flow mla.md MLA makes each entry smaller in a different way. Instead of storing keys and values per head, an MLA layer caches one compressed vector per token, the latent c