cd /news/ai-infrastructure/beyond-gqa-characterizing-modern-att… · home › topics › ai-infrastructure › article
[ARTICLE · art-173141] src=servingstudio.cs.washington.edu ↗ pub= topic=ai-infrastructure verified=true sentiment=· neutral

Beyond GQA: Characterizing Modern Attention Variants with ServingStudio

A ServingStudio analysis of eleven recent open models found that at a 128K-token context, newer attention designs need up to 139× less attention state per request, load up to 250× fewer bytes per decode step, and require up to 183× fewer prefill attention FLOPs than Qwen3-Coder-480B-A35B, a grouped-query attention (GQA) baseline of similar size. DeepSeek V4.1 Flash delivered the largest memory and decode gains, with attention running up to 44× faster in decode and up to 46× faster in prefill, while Qwen3.8 Flash Next posted the 183× prefill FLOP reduction. The authors conclude that bottlenecks and optimization opportunities identified in GQA models do not always carry over, and that inference research should study the new architectures — latent caches, sliding windows, sparse attention, compressed caches and linear attention — directly.

recommendation

Re-profile memory, bytes per decode step and prefill FLOPs on the specific attention variant you serve rather than reusing GQA baselines; the source measured up to 139× less attention state and 250× fewer decode bytes at 128K tokens.

more in playbook: Inference and self-hosting
read67 min views2 publishedNov 20, 2026

All posts Top: the eleven attention designs in today's open models. Bottom: the best new model on each measure, against the GQA baseline at 128K tokens.

Introduction #

Much of today's performance analysis of LLM inference still rests on models with grouped-query attention (GQA), in which multiple Query vectors attend to the same Key-Value vector to save memory. That analysis tells us when a step is bound by attention and when by the FFN, how many requests fit in memory, and which kernels are worth tuning. Today's open models, however, have moved beyond GQA. They combine latent caches, sliding windows, sparse attention, compressed caches and linear attention. These designs drastically change the HBM consumption, the memory , and the TFLOPs spent on the attention.

In this post, we look at eleven recent models: DeepSeek V4.1 Flash, DeepSeek V4 Pro, GLM-5.3-Flash, Qwen3.8 Flash Next, MiMo V2.6 Flash, Inkling, GLM-5.3, MiniMax-M3.1 Flash, Kimi K3, Kimi K3.1 Pro and Qwen3.8 27B. We compare them with Qwen3-Coder-480B-A35B, a GQA model of similar size to most of them.

At a 128K-token context, the new models need far less than the baseline:

  • Memory. A request keeps up to 139× less attention state (DeepSeek V4.1 Flash).

  • Decode. Each step loads up to 250× fewer bytes, and its attention runs up to 44× faster (DeepSeek V4.1 Flash).

  • Prefill. A prompt needs up to 183× fewer attention FLOPs (Qwen3.8 Flash Next), and its attention runs up to

    46× faster (DeepSeek V4.1 Flash). These changes matter beyond the numbers themselves: they change the answers that the analysis of GQA models gives.

Our main takeaway

The attention landscape has changed, and inference research needs to catch up.

New attention designs change how much memory a request needs, how many bytes each step moves, and how much it computes. As a result, the bottlenecks and optimization opportunities of GQA models do not always carry over. Future inference research should study these new models directly, instead of reusing conclusions drawn from GQA models.

Architectures #

All models except Qwen3.8 27B are mixture-of-experts (MoE) transformers. Their attention layers differ. The baseline uses GQA in every layer. GLM-5.3 replaces GQA with DSA in every layer, and MiniMax-M3.1 Flash replaces it with MSA. The other nine models mix two or more kinds of attention layers. The table below lists their size and attention choices.

Figure data as text: The models and their attention layers #

Models | Model | Role | Facts | Attention layers |

|---|---|---|---|
| Qwen3-Coder-480B-A35B | Baseline | 480B total; 35B active; 262K context | Grouped-query attention (GQA) ×62 (Full attention) | 
| MiMo V2.6 Flash | — | 309B total; 15B active; 1M context | Sliding-window attention (SWA) ×39 (Sliding window); Grouped-query attention (GQA) ×9 (Full attention) | 
| Inkling | — | 975B total; 41B active; 1M context | Sliding-window attention (SWA) ×55 (Sliding window); Grouped-query attention (GQA) ×11 (Full attention) | 
| GLM-5.3 | — | 753B total; ~40B active; 1M context | DeepSeek Sparse Attention (DSA) ×78 (Sparse attention) | 
| MiniMax-M3.1 Flash | — | ~428B total; ~23B active; 1M context | MiniMax Sparse Attention (MSA) ×60 (Sparse attention) | 
| DeepSeek V4 Pro | — | 1.6T total; 49B active; 1M context | Compressed Sparse Attention (CSA) ×30 (Sparse attention); Heavily Compressed Attention (HCA) ×31 (Full attention) | 

| DeepSeek V4.1 Flash | — | 552B backbone; 8B active in prefill; 16B active in decode; 1M context | Sliding-window attention (SWA) ×40 (Sliding window); Compressed Sparse Attention (CSA2) ×38 (Sparse attention) |

| Kimi K3 | — | 2.8T total; 104B active; 1M context | Kimi Delta Attention (KDA) ×69 (Linear attention); Gated MLA ×24 (Full attention) | 
| Kimi K3.1 Pro | — | 2.8T total; 104B active; 1M context | Kimi Delta Attention (KDA) ×69 (Linear attention); Gated DSA ×24 (Sparse attention) | 
| GLM-5.3-Flash | — | 320B total; 18B active; 1M context | Kimi Delta Attention (KDA) ×34 (Linear attention); DeepSeek Sparse Attention (DSA) ×11 (Sparse attention) | 
| Qwen3.8 Flash Next | — | 125B total; 6B active; 51B n-gram embedding; 262K context, 1M with YaRN | Gated DeltaNet (GDN) ×36 (Linear attention); Qwen Sparse Attention (QSA) ×12 (Sparse attention) | 
| Qwen3.8 27B | — | 27B dense; 262K context, 1M with YaRN | Gated DeltaNet (GDN) ×48 (Linear attention); Gated attention (GQA) ×16 (Full attention) | 

All of this figure's data (every setting and operation): data/model_cards.md

All analysis below assumes the model is deployed on B200 GPUs with data-parallel (DP) attention: each GPU runs every attention head of a layer, and each request stays on one GPU. We borrow kernels from vLLM or SGLang and measure them with ServingStudio.

Attention Variants #

We first go through how each attention design works. They fall into two branches. The first keeps an entry for every earlier token in the cache. On this branch, GQA and MLA make each entry smaller, while sliding windows, DSA, CSA and HCA cut how much of the cache is kept or read. The second branch, linear attention, folds the whole context into a state of fixed size.

### Grouped-query attention (GQA)

## Figure data as text: GQA in one layer of Qwen3-Coder-480B-A35B

Prefill: data flow

  • Main row: x (N × 6,144) → Projection → QK-norm → RoPE → Scores → Causal mask → Softmax → Weighted sum → Output projection → out (to the MoE FFN)
  • Write: RoPE (k) and Projection (v (no norm, no RoPE)) → Write K, V → KV cache, kept for the whole request
  • Scores reads K from the cache: reads K: L earlier + N in chunk
  • Weighted sum reads V from the cache: reads V: L earlier + N in chunk

Decode: data flow

- Main row: x (B × 6,144) → Projection → QK-norm → RoPE → Scores → Softmax → Weighted sum → Output projection → out (to the MoE FFN)
- Write: RoPE (k) and Projection (v (no norm, no RoPE)) → Write K, V → KV cache, kept for the whole request
- Scores reads K from the cache: reads K: all L tokens
  • Weighted sum reads V from the cache: reads V: all L tokens

Cache

- Prefill: KV cache, kept for the whole request; rows: K, V
- Decode: KV cache, kept for the whole request; rows: K, V
- 8 KV heads × 128 × (K + V) = 2 KiB per token and layer in FP8; 124 KiB per token over 62 layers

All of this figure's data (every setting and operation): [data/flow_gqa.md](https://servingstudio.cs.washington.edu/blog/attention-variants/assets/interactive/data/flow_gqa.md)

In standard multi-head attention, every query head has its own key head and value head. GQA lets a group of query heads share one key-value head, so each token keeps fewer keys and values. Our baseline, Qwen3-Coder-480B-A35B, has 96 query heads and 8 key-value heads in each layer, so 12 query heads share each key-value head.

The token's hidden state is first projected into queries, keys and values, and QK-norm and RoPE normalize the queries and keys and encode their positions. The keys and values are stored in the KV cache, where they stay for the entire lifecycle of the request. To compute its output, a new token compares its queries with the keys of all earlier tokens, turns these scores into weights with a softmax, and takes the weighted sum of the earlier tokens' values. The two blue arrows show these two reads from the cache: computing the scores reads every key, and the weighted sum reads every value.

Multi-head latent attention (MLA)

Figure data as text: Gated MLA in one layer of Kimi K3 #

Prefill: data flow

  • Main row: x (N × 7,168) → Down-projection → RMSNorm → Query up-projection → Scores → Causal mask → Softmax → Weighted sum → Output gate → Output projection → out (to the MoE FFN)
- Parallel branch: from x (x) → Gate projection → joins Output gate (g)
- Parallel branch: from Query up-projection → Expand K, V → joins Scores (K)
  • V goes from Expand K, V straight to Weighted sum
  • Write: RMSNorm (c_kv) and Down-projection (k_s, not normalized) → Write c_kv, k_s → Latent cache, kept for the whole request
  • Expand K, V reads c_kv, k_s from the cache: reads c_kv, k_s: L earlier + N in chunk

Decode: data flow

  • Main row: x (B × 7,168) → Down-projection → RMSNorm → Query up-projection → Absorb W_UK → Scores → Softmax → Weighted sum → Absorb W_UV → Output gate → Output projection → out (to the MoE FFN)
  • Parallel branch: from x (x) → Gate projection → joins Output gate (g)
  • q_s goes from Query up-projection straight to Scores
  • Write: RMSNorm (c_kv) and Down-projection (k_s, not normalized) → Write c_kv, k_s → Latent cache, kept for the whole request
  • Scores reads c_kv, k_s from the cache: reads c_kv, k_s: all L tokens
  • Weighted sum reads c_kv, k_s from the cache: same c_kv, as the values

Cache

- Prefill: Latent cache, kept for the whole request; rows: c_kv, k_s
- Decode: Latent cache, kept for the whole request; rows: c_kv, k_s
- c_kv (512 numbers) and k_s (64), read by all heads: 576 B per token and layer in FP8; 13.5 KiB over 24 layers

All of this figure's data (every setting and operation): [data/flow_mla.md](https://servingstudio.cs.washington.edu/blog/attention-variants/assets/interactive/data/flow_mla.md)

MLA makes each entry smaller in a different way. Instead of storing keys and values per head, an MLA layer caches one compressed vector per token, the latent c<sub>kv</sub>, which all query heads share. Each head can rebuild its own key and value

from c<sub>kv</sub> with two learned matrices, W<sub>UK</sub> and W<sub>UV</sub>. In prefill, the engine rebuilds the keys and values of the earlier tokens from their cached latents with W<sub>UK</sub> and W<sub>UV</sub> (Expand K, V), and then computes attention as in GQA. The rebuild is paid once and shared by the thousands of tokens in the chunk.

In decode, only one new token arrives at each step, so rebuilding every earlier key and value just for it would be wasteful. Instead, the engine folds W<sub>UK</sub> into the new token's query (Absorb W<sub>UK</sub>) and applies W<sub>UV</sub> after the weighted sum (Absorb W<sub>UV</sub>). Attention then works on the cached latents directly, so both blue arrows start from the same cache row.

Absorbing W<sub>UK</sub> does not work if RoPE rotates the keys, because the rotation depends on each token's position. The original MLA in DeepSeek V2 therefore caches one more small key part per token, k<sub>s</sub>, which all heads share, and

applies RoPE only to k<sub>s</sub>. Kimi K3, whose layer the figure shows, changes MLA in two small ways. It drops RoPE, so k<sub>s</sub> is a plain key that carries no position. It also adds an output gate. A separate projection of the input gives a gate g, and before the output projection each number of the attention output is scaled by sigmoid(g).

Sliding-window attention

Figure data as text: Sliding window in one window layer of MiMo V2.6 Flash #

Costs are for one whole layer on one GPU, as in the other figures.

Prefill: data flow

  • Main row: x (N × 4,096) → Projection → RoPE → Scores → Window mask → Softmax → Weighted sum → Output projection → out (to the MoE FFN)
  • Write: RoPE (k, v) → Write K, V → Window cache: only the last 128 tokens
  • Scores reads K from the cache: reads K: last 127 + N in chunk
  • Weighted sum reads V from the cache: reads V: last 127 + N in chunk

Decode: data flow

  • Main row: x (B × 4,096) → Projection → RoPE → Scores → Softmax → Weighted sum → Output projection → out (to the MoE FFN)
  • Write: RoPE (k, v) → Write K, V → Window cache: only the last 128 tokens
  • Scores reads K from the cache: reads K: last 127 + this token
  • Weighted sum reads V from the cache: reads V: last 127 + this token

Cache

- Prefill: Window cache: only the last 128 tokens; rows: K, V
- Decode: Window cache: only the last 128 tokens; rows: K, V
  • 8 KV heads × (192 + 128) = 2.5 KiB per token and layer in FP8; 320 KiB per layer at most, 12 MiB over 39 window layers

All of this figure's data (every setting and operation): data/flow_swa.md A sliding-window layer attends only to the last W tokens. Older tokens are never read again, so the cache keeps only W entries and stops growing once the context passes the window. The window can sit on top of either GQA or MLA. MiMo V2.6 Flash uses GQA-based windows in most of its layers, and DeepSeek V4.1 Flash uses MLA-based windows in all of its layers. Both use W = 128.

In prefill, each token in the chunk compares its query only with the last W tokens up to itself, so the scores form a narrow band along the diagonal. In decode, each step adds the new token's entry and drops the oldest one.

Inkling also mixes window layers, with W = 512, and GQA layers. Instead of RoPE, it encodes positions with a learned bias that it adds to each score, based on how far apart the two tokens are. Its attention kernel must therefore compute this bias as well.

DeepSeek Sparse Attention (DSA)

Figure data as text: Gated DSA in one layer of Kimi K3.1 Pro #

Prefill: data flow

  • Main row: x (N × 7,168) → Down-projection → Norms → Query up-projection → Absorb W_UK → Sparse scores → Softmax → Weighted sum → Absorb W_UV → Output gate → Output projection → out (to the MoE FFN)
- Parallel branch: from x (x) → Gate projection → joins Output gate (g)
- Parallel branch: from Query up-projection (q_I) → Index scores → Top-k → joins Sparse scores (2,048 selected)
  • w goes from Down-projection straight to Index scores
  • q_s goes from Query up-projection straight to Sparse scores
  • Write: Norms (k_I, c_kv) and Down-projection (k_s, not normalized) → Write caches → Two caches, kept for the whole request
  • Index scores reads k_I from the cache: reads k_I: L earlier + N in chunk
  • Sparse scores reads c_kv, k_s from the cache: reads only the 2,048 selected rows
  • Weighted sum reads c_kv, k_s from the cache: same rows, as the values

Decode: data flow

  • Main row: x (B × 7,168) → Down-projection → Norms → Query up-projection → Absorb W_UK → Sparse scores → Softmax → Weighted sum → Absorb W_UV → Output gate → Output projection → out (to the MoE FFN)
- Parallel branch: from x (x) → Gate projection → joins Output gate (g)
- Parallel branch: from Query up-projection (q_I) → Index scores → Top-k → joins Sparse scores (2,048 selected)
  • w goes from Down-projection straight to Index scores
  • q_s goes from Query up-projection straight to Sparse scores
  • Write: Norms (k_I, c_kv) and Down-projection (k_s, not normalized) → Write caches → Two caches, kept for the whole request
  • Index scores reads k_I from the cache: reads every k_I: all L tokens
  • Sparse scores reads c_kv, k_s from the cache: reads only the 2,048 selected rows
  • Weighted sum reads c_kv, k_s from the cache: same rows, as the values

Cache

- Prefill: Two caches, kept for the whole request; rows: k_I, c_kv, k_s
- Decode: Two caches, kept for the whole request; rows: k_I, c_kv, k_s
  • k_I: 128 FP8 + 4 B scale = 132 B; c_kv 512 FP8 + 16 B scales + k_s 64 BF16 = 656 B; 788 B per token and layer, 18.5 KiB over 24 layers

All of this figure's data (every setting and operation): data/flow_dsa.md Not all earlier tokens matter equally. After the softmax, most of the weight usually falls on a few of them, and reading the rest barely changes the output. DSA, the sparse attention that DeepSeek introduced in V3.2, adds a small indexer in front of MLA to find these tokens. The indexer keeps its own short key, k<sub>I</sub>, for every token. For each new token, it scores every earlier token with a cheap dot product and keeps the 2,048 tokens with the highest scores. The layer then runs ordinary MLA over only these selected tokens.

The indexer still scores the whole context, but its key is small, so each score is cheap. Sparse attention therefore cuts what each step reads and computes, but not what a request stores: any token may be selected later, so every token stays in the cache.

In prefill, different tokens select different earlier tokens. Expanding the whole context would rebuild keys and values that most tokens never read, so vLLM keeps long Kimi K3.1 Pro prompts on the absorbed path in prefill too.

GLM-5.3 and GLM-5.3-Flash also use DSA, each with one change to the indexer. In GLM-5.3, only some layers run an indexer, and the others reuse the most recent selection. GLM-5.3-Flash keeps one index key for every 4 tokens and selects tokens in groups of 4.

MiniMax/Qwen Sparse Attention (MSA/QSA)

Figure data as text: MSA in one layer of MiniMax-M3.1 Flash #

Costs are for one whole layer on one GPU, as in the other figures.

Prefill: data flow

- Main row: x (N × 6,144) → Projection → Norms, RoPE → Sparse attention → Output projection → out (to the MoE FFN)
- Parallel branch: from Norms, RoPE (q_I) → Index scores → Top-k → joins Sparse attention (16 blocks per group)
- Write: Norms, RoPE (k, k_I) and Projection (v (no norm, no RoPE)) → Write caches → Two caches, kept for the whole request
  • Index scores reads k_I from the cache: reads k_I: L earlier + N in chunk
  • Sparse attention reads K, V from the cache: reads 16 blocks per group

Decode: data flow

- Main row: x (B × 6,144) → Projection → Norms, RoPE → Sparse attention → Output projection → out (to the MoE FFN)
- Parallel branch: from Norms, RoPE (q_I) → Index scores → Top-k → joins Sparse attention (16 blocks per group)
- Write: Norms, RoPE (k, k_I) and Projection (v (no norm, no RoPE)) → Write caches → Two caches, kept for the whole request
  • Index scores reads k_I from the cache: reads every k_I: all L tokens
  • Sparse attention reads K, V from the cache: reads 16 blocks per group

Cache

- Prefill: Two caches, kept for the whole request; rows: k_I, K, V
- Decode: Two caches, kept for the whole request; rows: k_I, K, V
- k_I: 128 FP8 = 128 B; K, V: 4 heads × 128 × 2 FP8 = 1,024 B; 1,152 B per token and layer, 67.5 KiB over 60 layers

All of this figure's data (every setting and operation): [data/flow_msa.md](https://servingstudio.cs.washington.edu/blog/attention-variants/assets/interactive/data/flow_msa.md)

MSA in MiniMax-M3.1 Flash builds sparse attention on GQA and selects whole blocks of 128 tokens instead of single tokens. Its indexer keeps one short key per token and scores every earlier token, as in DSA. A block's score is the highest score of any token in it. Each group of query heads picks its own 16 blocks, and the block holding the newest tokens is always among them.

Figure data as text: QSA in one layer of Qwen3.8 Flash Next #

Costs are for one whole layer on one GPU, as in the other figures.

Prefill: data flow

- Main row: x (N × 2,560) → Projection → Norms, RoPE → Sparse attention → Output projection → out (to the MoE FFN)
- Parallel branch: from Projection (q_I, k_raw) → Index prep → Index scores → Top-k → joins Sparse attention (2,048 selected)
  • gate, v goes from Projection straight to Sparse attention
  • Write: Index prep (k̄, k_raw) → Write index → Two caches, kept for the whole request
  • Index scores reads k̄ from the cache: reads every k̄: (L+N)/4
  • Sparse attention reads K, V from the cache: appends k, v; reads 2,048 selected

Decode: data flow

- Main row: x (B × 2,560) → Projection → Norms, RoPE → Sparse attention → Output projection → out (to the MoE FFN)
- Parallel branch: from Projection (q_I, k_raw) → Index prep → Index scores → Top-k → joins Sparse attention (2,048 selected)
  • gate, v goes from Projection straight to Sparse attention
  • Write: Index prep (k̄, k_raw) → Write index → Two caches, kept for the whole request
  • Index scores reads k̄ from the cache: reads every k̄: L/4
  • Sparse attention reads K, V from the cache: appends k, v; reads 2,048 selected

Cache

- Prefill: Two caches, kept for the whole request; rows: k̄, K, V
- Decode: Two caches, kept for the whole request; rows: k̄, K, V
  • k̄: one 128 B key (FP8) per 4 tokens; K, V: 2 heads × 256 × 2 FP8 = 1,024 B per token and layer. Plus a ring of the last raw keys, 1 KiB per request and layer. 12 QSA layers.

All of this figure's data (every setting and operation): data/flow_qsa.md QSA in Qwen3.8 Flash Next also builds on GQA, but its blocks hold only 4 tokens. It also shrinks the index: the indexer keeps one key for every block of 4 tokens instead of one per token, so scanning the whole context reads much less.

Compressed sparse attention (CSA)

Figure data as text: Compressed and shared caches in layer 2 of DeepSeek V4.1 Flash #

Costs are for one whole layer on one GPU, as in the other figures.

Prefill: data flow

  • Main row: x (N × 5,120) → Projection → Norms → Query up-projection → q RoPE → Attention over the window and the selected entries → Output projection → out (to the MoE FFN)
- Parallel branch: from x (x) → Side-stream projections → Compress → Index key projection
- Parallel branch: from q RoPE (q_I) → Index scores → Top-k → joins Attention over the window and the selected entries (512 selected)
  • w goes from Side-stream projections straight to Index scores
  • kv goes from Norms straight to q RoPE
  • Write: Index key projection (c, k_c) and q RoPE (kv) → Write caches → Caches of layer 2; layers 3–7 read its compressed rows
  • Index scores reads index keys from the cache: reads all index keys
  • Attention over the window and the selected entries reads compressed from the cache
  • Attention over the window and the selected entries reads window from the cache: reads 512 entries + window per query

Decode: data flow

  • Main row: x (B × 5,120) → Projection → Norms → Query up-projection → q RoPE → Attention over the window and the selected entries → Output projection → out (to the MoE FFN)
- Parallel branch: from x (x) → Side-stream projections → Compress → Index key projection
- Parallel branch: from q RoPE (q_I) → Index scores → Top-k → joins Attention over the window and the selected entries (512 selected)
  • w goes from Side-stream projections straight to Index scores
  • kv goes from Norms straight to q RoPE
  • Write: Index key projection (c, k_c) and q RoPE (kv) → Write caches → Caches of layer 2; layers 3–7 read its compressed rows
  • Index scores reads index keys from the cache: reads all index keys
  • Attention over the window and the selected entries reads compressed from the cache
  • Attention over the window and the selected entries reads window from the cache: reads 512 entries + 128 in window

Cache

  • Prefill: Caches of layer 2; layers 3–7 read its compressed rows; rows: index keys, compressed, window

  • Decode: Caches of layer 2; layers 3–7 read its compressed rows; rows: index keys, compressed, window

  • Per 2 tokens: one 288 B entry (NVFP4) and one 68 B index key (FP4), kept for the whole request. Window: 128 × 528 B (MXFP8) All of this figure's data (every setting and operation): data/flow_compressed.md

DeepSeek V4 introduced CSA, which combines the ideas above. The most recent 128 tokens are kept in a window, as in SWA. Older tokens are compressed: a learned compressor merges a few consecutive tokens into one entry, 4 tokens in DeepSeek V4 Pro. An indexer, as in DSA, then selects some of these entries, 1,024 in DeepSeek V4 Pro, and attention reads only the window and the selected entries. Unlike MLA, the cached vector itself serves as the key and the value

of every head, with no W<sub>UK</sub> or W<sub>UV</sub>. DeepSeek V4 Pro pairs CSA with heavily compressed attention (HCA) and alternates the two across its layers. An HCA layer merges every 128 tokens into one entry, so its compressed cache is small enough to read in full. It therefore needs no indexer: attention reads the window and every compressed entry.

DeepSeek V4.1 Flash uses a revised version, which we call CSA2. It compresses less: layers in the first half merge every 2 tokens, and layers in the second half do not compress at all. To save memory, it stores the entries in FP4 and lets several layers share one cache and one selection. Only 4 of the 40 layers write a compressed cache, and the other layers read the cache of the nearest writer before them. The indexer runs in 8 layers: the 4 writers, and 4 more in the second half that select again from layer 20's cache. Every other layer reuses the selection of the nearest indexer layer before it. The figure follows layer 2, the first writer. Layers 3 to 7 read its cache and reuse its selection.

Layer 20 writes the cache that all later layers read, so layers 21 to 39 write no compressed cache. In prefill, these layers therefore only need to run over the last 128 tokens of the prompt, enough to fill their SWA windows. DeepSeek calls this SWA bounded replay, and vLLM (PR #58132) and SGLang implement it.

Gated DeltaNet / Kimi Delta Attention (GDN/KDA)

Figure data as text: Kimi Delta Attention in one linear layer of GLM-5.3-Flash #

Costs are for one whole layer on one GPU, as in the other figures.

Prefill: data flow

  • Main row: x (N × 4,096) → Projection → Short conv → Per-block preparation → State pass, block by block → Per-block read-out → Gated norm → Output projection → out (to the MoE FFN)
- Parallel branch: from Projection (f_a, g_a, b) → Gates → joins Per-block preparation (f, b), Delta-rule update and read-out (f), Gated norm (g)
- q goes from Short conv straight to Per-block read-out
- Q Kᵀ goes from Per-block preparation straight to Per-block read-out
- Write: Short conv (last 3 inputs) → Write conv → State, fixed in size
- Write: State pass, block by block (S) → Write S → State, fixed in size
  • Short conv reads conv from the cache: pads with last 3
  • State pass, block by block reads S from the cache: reads S once

Decode: data flow

- Main row: x (B × 4,096) → Projection → Short conv → Delta-rule update and read-out → Gated norm → Output projection → out (to the MoE FFN)
- Parallel branch: from Projection (f_a, g_a) → Gates → joins Per-block preparation (f, b), Delta-rule update and read-out (f), Gated norm (g)
  • b goes from Projection straight to Delta-rule update and read-out
- Write: Short conv (inputs) → Write conv → State, fixed in size
- Write: Delta-rule update and read-out (S) → Write S → State, fixed in size
  • Short conv reads conv from the cache: reads last 3

  • Delta-rule update and read-out reads S from the cache: reads S Cache

- Prefill: State, fixed in size; conv, 144 KiB: the prefix's last 3 pad the chunk; its last 3 replace them; S: 64 × 128 × 128 FP32 = 4 MiB; read once, written once
- Decode: State, fixed in size; conv: last 3 inputs, 3 × 24,576 BF16 = 144 KiB; S: 64 heads × 128 × 128, FP32 = 4 MiB
  • 4.1 MiB per layer, whatever L is; 141 MiB per request over 34 KDA layers

All of this figure's data (every setting and operation): data/flow_linear.md

A linear-attention layer keeps no entry per token. Instead, each head keeps one matrix S of fixed size, with one row per key channel and one column per value channel. A token writes into S by adding the product k v<sup>⊤</sup> of its key and value, and reads its output as o = S<sup>⊤</sup> q. Because every token adds to the same matrix, the state never grows. But it also never forgets, so over a long context the stored values pile up and blur into each other.

Gated DeltaNet (GDN), used by Qwen3.8 Flash Next and Qwen3.8 27B, fixes this in two ways, and each gives it part of its name. The first is the delta rule. Before writing, the token reads what S already stores for its key, S<sup>⊤</sup> k, and writes only the difference from its own value v. With a write strength β between 0 and 1, the update is S ← S + β k (v - S<sup>⊤</sup> k)<sup>⊤</sup>. A key that is written again therefore replaces its old value instead of adding to it. The second is the gate. Before each write, S is multiplied by a decay α between 0 and 1, so content that is not written again fades over time. In GDN, α is one number per head, so the whole state of a head fades at one rate.

Kimi Delta Attention (KDA), used by GLM-5.3-Flash, Kimi K3 and Kimi K3.1 Pro, keeps the delta rule but makes the gate finer. Its α holds one decay per key channel, so each row of S fades at its own rate: some rows keep old content for long, while others forget quickly. The figure shows a KDA layer. Its Gates step turns each token into the input of the decay α and into an output gate g. The attention kernel then computes α and applies the delta rule, and g scales the attention output before the output projection, as in Kimi K3's MLA layer.

In decode, each step reads and writes the same state at any context length. In prefill, running the update token by token would leave the GPU idle. The kernel therefore splits the prompt into chunks of 64 tokens, computes each chunk with matrix multiplies, and passes S from chunk to chunk. A state of fixed size cannot keep every detail of a long context, so these models keep a few full-attention, DSA or QSA layers to look back exactly.

Takeaway. Each variant in one line:

  • GQA: several query heads share one key-value head.
  • MLA: each token's keys and values are compressed into one latent vector.
  • SWA: the layer keeps only the lastW tokens.
  • DSA: an indexer selects the top-k tokens, and MLA runs over only those.
  • MSA: an indexer selects the top-k blocks of tokens, and GQA runs over only those.
  • QSA: like MSA, with much smaller blocks, so the selection is finer.
  • CSA: the keys of several tokens merge into one entry, and DSA runs over these entries.
  • HCA: many more tokens merge into one entry, so attention reads all entries without an indexer.
  • CSA2: a few layers write the compressed keys, and the other layers share them.
  • GDN: every token updates a fixed-size state whose old content keeps fading.
  • KDA: GDN in which each channel of the state fades at its own rate.

Attention Memory per Request #

Each request keeps its attention state in GPU memory, and this memory limits how many requests a GPU can serve at once. With DP attention, all of a request's attention state sits on one GPU. We count every KV cache in FP8, one byte per number, unless the model itself uses FP4, as DeepSeek V4.1 Flash does. Kimi K3.1 Pro is the one exception: the sparse MLA kernel that vLLM runs for its 96 heads keeps k<sub>s</sub> in BF16. Linear-attention states and DeepSeek V4 Pro's compressor states are kept in FP32. You can explore every layout

on the Models page. The first figure below splits one request's memory into its parts, at the context set by the slider. The second lists the total for each model at several contexts, colored from blue for little memory to orange-red for much. Qwen3-Coder-480B-A35B does not support contexts beyond 256K tokens, so its 1M cell is empty.

Figure data as text: Attention memory of one request #

Total per model by context length

Model 4K 8K 16K 32K 64K 128K 256K 512K 1M
Qwen3-Coder-480B-A35B 496 MiB 992 MiB 1.94 GiB 3.88 GiB 7.75 GiB 15.5 GiB 31.0 GiB not supported beyond 256K not supported beyond 256K
MiMo V2.6 Flash 57.2 MiB 102 MiB 192 MiB 372 MiB 732 MiB 1.42 GiB 2.82 GiB 5.64 GiB 11.3 GiB
Inkling 206 MiB 294 MiB 470 MiB 822 MiB 1.49 GiB 2.87 GiB 5.62 GiB 11.1 GiB 22.1 GiB
GLM-5.3 186 MiB 373 MiB 745 MiB 1.46 GiB 2.91 GiB 5.82 GiB 11.6 GiB 23.3 GiB 46.6 GiB
MiniMax-M3.1 Flash 270 MiB 540 MiB 1.05 GiB 2.11 GiB 4.22 GiB 8.44 GiB 16.9 GiB 33.8 GiB 67.5 GiB
DeepSeek V4 Pro 43.7 MiB 65.3 MiB 108 MiB 194 MiB 367 MiB 711 MiB 1.37 GiB 2.71 GiB 5.40 GiB
DeepSeek V4.1 Flash 6.15 MiB 9.63 MiB 16.6 MiB 30.5 MiB 58.3 MiB 114 MiB 225 MiB 448 MiB 893 MiB
Kimi K3 483 MiB 537 MiB 645 MiB 861 MiB 1.26 GiB 2.11 GiB 3.79 GiB 7.17 GiB 13.9 GiB
Kimi K3.1 Pro 502 MiB 576 MiB 724 MiB 1020 MiB 1.57 GiB 2.73 GiB 5.04 GiB 9.65 GiB 18.9 GiB
GLM-5.3-Flash 164 MiB 188 MiB 234 MiB 328 MiB 515 MiB 890 MiB 1.60 GiB 3.06 GiB 5.99 GiB
Qwen3.8 Flash Next 160 MiB 209 MiB 308 MiB 506 MiB 902 MiB 1.65 GiB 3.20 GiB 6.30 GiB 12.5 GiB
Qwen3.8 27B 275 MiB 403 MiB 659 MiB 1.14 GiB 2.14 GiB 4.14 GiB 8.14 GiB 16.1 GiB 32.1 GiB

Takeaways the figure shows as the slider moves

  • 4K–16K: At short contexts, the fixed linear-attention states dominate. At 4K tokens, Kimi K3 and Kimi K3.1 Pro keep 483 MiB and 502 MiB per request, about as much as the baseline's 496 MiB.
  • 32K–64K: As the context grows, the per-token caches catch up with the fixed states, and the bytes per token start to decide.
  • 128K–256K: At 128K tokens, DeepSeek V4.1 Flash keeps 139× less than the baseline's 15.5 GiB. Even MiniMax-M3.1 Flash, which keeps the most of the new models, keeps 1.8× less.
  • 512K–1M: Past 256K tokens, only the new models go on. At 1M tokens, they keep from 893 MiB in DeepSeek V4.1 Flash to 67.5 GiB in MiniMax-M3.1 Flash.

All of this figure's data (every setting and operation): data/kv_storage.md

Figure data as text: Attention memory of one request as the context grows #

Per model by context length

Model 4K 8K 16K 32K 64K 128K 256K 512K 1M
Qwen3-Coder-480B-A35B 496 MiB 992 MiB 1.94 GiB 3.88 GiB 7.75 GiB 15.5 GiB 31.0 GiB not supported beyond 256K not supported beyond 256K
MiMo V2.6 Flash 57.2 MiB 102 MiB 192 MiB 372 MiB 732 MiB 1.42 GiB 2.82 GiB 5.64 GiB 11.3 GiB
Inkling 206 MiB 294 MiB 470 MiB 822 MiB 1.49 GiB 2.87 GiB 5.62 GiB 11.1 GiB 22.1 GiB
GLM-5.3 186 MiB 373 MiB 745 MiB 1.46 GiB 2.91 GiB 5.82 GiB 11.6 GiB 23.3 GiB 46.6 GiB
MiniMax-M3.1 Flash 270 MiB 540 MiB 1.05 GiB 2.11 GiB 4.22 GiB 8.44 GiB 16.9 GiB 33.8 GiB 67.5 GiB
DeepSeek V4 Pro 43.7 MiB 65.3 MiB 108 MiB 194 MiB 367 MiB 711 MiB 1.37 GiB 2.71 GiB 5.40 GiB
DeepSeek V4.1 Flash 6.15 MiB 9.63 MiB 16.6 MiB 30.5 MiB 58.3 MiB 114 MiB 225 MiB 448 MiB 893 MiB
Kimi K3 483 MiB 537 MiB 645 MiB 861 MiB 1.26 GiB 2.11 GiB 3.79 GiB 7.17 GiB 13.9 GiB
Kimi K3.1 Pro 502 MiB 576 MiB 724 MiB 1020 MiB 1.57 GiB 2.73 GiB 5.04 GiB 9.65 GiB 18.9 GiB
GLM-5.3-Flash 164 MiB 188 MiB 234 MiB 328 MiB 515 MiB 890 MiB 1.60 GiB 3.06 GiB 5.99 GiB
Qwen3.8 Flash Next 160 MiB 209 MiB 308 MiB 506 MiB 902 MiB 1.65 GiB 3.20 GiB 6.30 GiB 12.5 GiB
Qwen3.8 27B 275 MiB 403 MiB 659 MiB 1.14 GiB 2.14 GiB 4.14 GiB 8.14 GiB 16.1 GiB 32.1 GiB

All of this figure's data (every setting and operation): data/kv_growth.md Next, we look at decode: how many bytes one decode step reads, and how long it takes.

Bytes read per decode step

We first count the bytes that one decode step reads for one request. The hatched bars in the first figure below show what each request keeps. While a full-attention or window layer reads all of its cache at every step, a sparse layer reads only its index and the entries it selects, a fraction of the KV cache storage. At 128K tokens, GLM-5.3 reads only 7.3% of what it keeps at each step, and MiniMax-M3.1 Flash reads 12.5%.

Figure data as text: Bytes one request reads per decode step #

Total per model by context length

Model 4K 8K 16K 32K 64K 128K 256K 512K 1M
Qwen3-Coder-480B-A35B 496 MiB 992 MiB 1.94 GiB 3.88 GiB 7.75 GiB 15.5 GiB 31.0 GiB not supported beyond 256K not supported beyond 256K
MiMo V2.6 Flash 57.2 MiB 102 MiB 192 MiB 372 MiB 732 MiB 1.42 GiB 2.82 GiB 5.64 GiB 11.3 GiB
Inkling 214 MiB 302 MiB 478 MiB 830 MiB 1.50 GiB 2.87 GiB 5.62 GiB 11.1 GiB 22.1 GiB
GLM-5.3 98.6 MiB 109 MiB 131 MiB 174 MiB 261 MiB 434 MiB 781 MiB 1.44 GiB 2.79 GiB
MiniMax-M3.1 Flash 150 MiB 180 MiB 240 MiB 360 MiB 600 MiB 1.05 GiB 1.99 GiB 3.87 GiB 7.62 GiB
DeepSeek V4 Pro 61.6 MiB 66.0 MiB 74.8 MiB 92.5 MiB 128 MiB 199 MiB 340 MiB 623 MiB 1.16 GiB
DeepSeek V4.1 Flash 9.84 MiB 11.6 MiB 15.0 MiB 21.9 MiB 35.7 MiB 63.4 MiB 119 MiB 229 MiB 450 MiB
Kimi K3 911 MiB 965 MiB 1.05 GiB 1.26 GiB 1.68 GiB 2.52 GiB 4.21 GiB 7.59 GiB 14.3 GiB
Kimi K3.1 Pro 900 MiB 913 MiB 937 MiB 987 MiB 1.06 GiB 1.25 GiB 1.64 GiB 2.41 GiB 3.96 GiB
GLM-5.3-Flash 294 MiB 295 MiB 298 MiB 304 MiB 315 MiB 338 MiB 383 MiB 474 MiB 656 MiB
Qwen3.8 Flash Next 246 MiB 248 MiB 251 MiB 257 MiB 269 MiB 293 MiB 341 MiB 437 MiB 629 MiB
Qwen3.8 27B 422 MiB 550 MiB 806 MiB 1.29 GiB 2.29 GiB 4.29 GiB 8.29 GiB 16.3 GiB 32.3 GiB

Takeaways the figure shows as the slider moves

  • 4K–16K: At short contexts, the linear-attention state is most of what a step reads, because each step reads it and writes it back at any context length. In GLM-5.3-Flash, it is over 90%.
  • 32K–128K: A sparse layer keeps every token but reads only its index and the selected entries. At 128K tokens, GLM-5.3 keeps 5.8 GiB per request, but a step reads only 434 MiB, 14× less, and 80% of that is the index.
  • 256K–1M: The index read keeps growing with the context. At 1M tokens, it is more than half of a step in every sparse model, and 98% in DeepSeek V4.1 Flash and MiniMax-M3.1 Flash. Kimi K3's dense MLA reads its whole cache instead, 3.6× more than Kimi K3.1 Pro.

All of this figure's data (every setting and operation): data/decode_bytes.md

Figure data as text: Bytes one request reads per decode step as the context grows #

Per model by context length

Model 4K 8K 16K 32K 64K 128K 256K 512K 1M
Qwen3-Coder-480B-A35B 496 MiB 992 MiB 1.94 GiB 3.88 GiB 7.75 GiB 15.5 GiB 31.0 GiB not supported beyond 256K not supported beyond 256K
MiMo V2.6 Flash 57.2 MiB 102 MiB 192 MiB 372 MiB 732 MiB 1.42 GiB 2.82 GiB 5.64 GiB 11.3 GiB
Inkling 214 MiB 302 MiB 478 MiB 830 MiB 1.50 GiB 2.87 GiB 5.62 GiB 11.1 GiB 22.1 GiB
GLM-5.3 98.6 MiB 109 MiB 131 MiB 174 MiB 261 MiB 434 MiB 781 MiB 1.44 GiB 2.79 GiB
MiniMax-M3.1 Flash 150 MiB 180 MiB 240 MiB 360 MiB 600 MiB 1.05 GiB 1.99 GiB 3.87 GiB 7.62 GiB
DeepSeek V4 Pro 61.6 MiB 66.0 MiB 74.8 MiB 92.5 MiB 128 MiB 199 MiB 340 MiB 623 MiB 1.16 GiB
DeepSeek V4.1 Flash 9.84 MiB 11.6 MiB 15.0 MiB 21.9 MiB 35.7 MiB 63.4 MiB 119 MiB 229 MiB 450 MiB
Kimi K3 911 MiB 965 MiB 1.05 GiB 1.26 GiB 1.68 GiB 2.52 GiB 4.21 GiB 7.59 GiB 14.3 GiB
Kimi K3.1 Pro 900 MiB 913 MiB 937 MiB 987 MiB 1.06 GiB 1.25 GiB 1.64 GiB 2.41 GiB 3.96 GiB
GLM-5.3-Flash 294 MiB 295 MiB 298 MiB 304 MiB 315 MiB 338 MiB 383 MiB 474 MiB 656 MiB
Qwen3.8 Flash Next 246 MiB 248 MiB 251 MiB 257 MiB 269 MiB 293 MiB 341 MiB 437 MiB 629 MiB
Qwen3.8 27B 422 MiB 550 MiB 806 MiB 1.29 GiB 2.29 GiB 4.29 GiB 8.29 GiB 16.3 GiB 32.3 GiB

All of this figure's data (every setting and operation): data/decode_bytes_grid.md

Attention time per decode step

We then measure the attention time of one decode step for a batch of 32 requests on one B200, varying the context length from 4K to 1M tokens. Each bar adds up the time of the attention kernels in every layer of the model, leaving out the FFN modules. Its colors show how much of that time each kind of attention takes: global, window, sparse or linear attention, and the indexer of the sparse layers. Gray segments add the rest of each attention module's work: the QKV and output projections, norms, RoPE and similar small kernels.

At long contexts, the caches of 32 requests no longer fit in one GPU's memory. No server would run such a batch, but we still measure its kernel time.

Figure data as text: Measured attention time of one decode step #

ms per decode step, whole model · B200, DP attention (all heads on each GPU). Hover a bar for the backend.

Total per model by context length (Batch 32, With other ops)

Model 4K 8K 16K 32K 64K 128K 256K 512K 1M
Qwen3-Coder-480B-A35B 5.82 ms 8.19 ms 12.9 ms 22.5 ms 41.1 ms 78.9 ms 154 ms not supported beyond 256K not supported beyond 256K
MiMo V2.6 Flash 2.81 ms 3.13 ms 3.89 ms 5.42 ms 8.46 ms 14.5 ms 26.6 ms 50.6 ms 96.3 ms
Inkling 8.77 ms 9.98 ms 12.6 ms 17.7 ms 27.3 ms 46.1 ms 83.6 ms 159 ms 307 ms
GLM-5.3 8.30 ms 8.38 ms 8.68 ms 8.96 ms 9.49 ms 10.5 ms 12.4 ms 16.2 ms 23.7 ms
MiniMax-M3.1 Flash 4.32 ms 4.44 ms 4.71 ms 5.38 ms 6.33 ms 8.62 ms 13.2 ms 22.5 ms 41.3 ms
DeepSeek V4 Pro 6.57 ms 6.57 ms 6.60 ms 6.92 ms 7.24 ms 7.87 ms 9.02 ms 11.2 ms 15.8 ms
DeepSeek V4.1 Flash 3.17 ms 3.19 ms 3.26 ms 3.36 ms 3.52 ms 3.83 ms 4.45 ms 5.67 ms 8.08 ms
Kimi K3 18.7 ms 19.0 ms 19.9 ms 21.7 ms 25.5 ms 31.1 ms 48.8 ms 79.4 ms 141 ms
Kimi K3.1 Pro 20.8 ms 20.9 ms 21.2 ms 21.5 ms 22.1 ms 23.3 ms 25.4 ms 29.8 ms 38.4 ms
GLM-5.3-Flash 6.01 ms 6.01 ms 6.04 ms 6.07 ms 6.18 ms 6.35 ms 6.61 ms 7.10 ms 8.16 ms
Qwen3.8 Flash Next 4.51 ms 4.52 ms 4.52 ms 4.56 ms 4.63 ms 4.78 ms 5.09 ms 5.56 ms 6.54 ms
Qwen3.8 27B 5.27 ms 5.87 ms 7.05 ms 9.40 ms 14.1 ms 23.5 ms 42.3 ms 79.9 ms 155 ms

Backend per model

Model Backend
Qwen3-Coder-480B-A35B FlashInfer trtllm-gen paged prefill/decode, FP8 Q/KV
MiMo V2.6 Flash vLLM FlashAttention 4, FP8 KV, with flash-attention PR #2918 (unpatched FA4 FP8 split-KV decode is wrong)
Inkling per cell the faster of vLLM Inkling fork tml_fa4 with a BF16 cache and SGLang FA4 with MXFP8 Q/K/V (block-scaled QK, in-kernel V dequant), both with the sheared relative bias; Triton fused_qkvr_prep (short conv + q/k norm + cache write)
GLM-5.3 FlashInfer trtllm-gen FP8 sparse MLA, DeepGEMM FP8 indexer, vLLM top-k (cooperative <= 64 rows)
MiniMax-M3.1 Flash vLLM MiniMax-M3 MSA with FP8 K/V and index keys; decode indexer: CuTe DSL index scoring + FlashInfer top-k; decode attention: Triton sparse GQA below 16 requests, CUTLASS fmha_sm100 from 16; prefill indexer: fmha_sm100 score-only (score + block max pool) + FlashInfer top-k; prefill attention: CuTe DSL KV-outer sparse attention + combine
DeepSeek V4 Pro vLLM FlashMLA sparse MLA on the fp8_ds_mla cache, all 128 heads: BF16 prefill (flash_mla_sparse_fwd), FP8 decode under CUDA graphs (Sm100 Head128 split-KV); CSA (C4) layers attend the 128-token window + top-1024 compressed entries, HCA (C128) layers the window + every compressed entry, one call per layer; DeepGEMM FP8 indexer logits, vLLM top-k (cooperative <= 64 rows, persistent above)
DeepSeek V4.1 Flash FlashMLA (compressed sparse MLA; CuTe DSL swap-AB decode for steps of <= 16 requests), DeepGEMM FP8 indexer, vLLM top-k; prefill with decoder bounded replay, as vLLM's V4.1 fork runs it by default
Kimi K3 FlashInfer cute-dsl MLA decode (FP8 q/KV, all 96 heads, no padding); TRT-LLM-gen ragged prefill (vLLM TRTLLM_RAGGED, FP8 Q/K/V); FlashKDA prefill and fused CUDA KDA decode as Kimi K3.1 Pro
Kimi K3.1 Pro vLLM FlashMLA sparse MLA on the fp8_ds_mla cache: FP8 decode (BF16 q padded to 128 heads); prefill upconverts the context to BF16 (BF16 sparse attention at 128 heads); DeepGEMM FP8 indexer, vLLM top-k (cooperative <= 64 rows); FlashKDA prefill, vLLM fused CUDA KDA decode (fused_kda_decode)
GLM-5.3-Flash FlashInfer trtllm-gen FP8 sparse MLA, DeepGEMM FP8 indexer, vLLM top-k (cooperative <= 64 rows); FlashKDA prefill, FLA Triton KDA decode
Qwen3.8 Flash Next vLLM Triton sparse GQA, vLLM top-k (cooperative <= 64 rows); FlashInfer / Triton GDN
Qwen3.8 27B FlashInfer trtllm-gen paged prefill/decode, FP8 Q/KV (FA4 has no FP8-KV path at head dim 256); FlashInfer CUTLASS GDN chunked prefill, vLLM Triton GDN decode

Takeaways the figure shows as the slider moves

  • 4K–16K, batch 32: At short contexts, fixed work decides. Kimi K3 and Kimi K3.1 Pro read and write their large linear-attention states at every step, so up to 8K tokens they are slower than the baseline. Inkling is slower too, because its 55 window layers take about 3.5 ms per step at any context.
  • 32K–128K, batch 32: The baseline's time grows with the bytes it reads, to 75.9 ms at 128K tokens. DeepSeek V4.1 Flash takes 1.7 ms there, 44× less.
  • 256K–1M, batch 32: At long contexts, the indexer takes most of the time in GLM-5.3 and MiniMax-M3.1 Flash, and the models with full-attention layers slow down the most. At 1M tokens, Qwen3.8 Flash Next takes 4.1 ms per step and Inkling 303 ms.
  • 4K–32K, batch 1: With one request, a step reads too little to keep the GPU busy. Up to 32K tokens, every model takes 0.3 to 2.0 ms in attention kernels.
  • 64K–256K, batch 1: The baseline's time grows with the context, to 5.4 ms at 256K tokens. The other new models stay at or below 2.3 ms, but Inkling reaches 3.5 ms, twice the time of Qwen3.8 27B, which reads more bytes per step.
  • 512K–1M, batch 1: Even at 1M tokens, GLM-5.3-Flash and Qwen3.8 Flash Next take less than 0.7 ms per step.

All of this figure's data (every setting and operation): data/decode_time.md

Figure data as text: Measured attention time of one decode step as the context grows #

Per model by context length (Batch 32, With other ops)

Model 4K 8K 16K 32K 64K 128K 256K 512K 1M
Qwen3-Coder-480B-A35B 5.82 ms 8.19 ms 12.9 ms 22.5 ms 41.1 ms 78.9 ms 154 ms not supported beyond 256K not supported beyond 256K
MiMo V2.6 Flash 2.81 ms 3.13 ms 3.89 ms 5.42 ms 8.46 ms 14.5 ms 26.6 ms 50.6 ms 96.3 ms
Inkling 8.77 ms 9.98 ms 12.6 ms 17.7 ms 27.3 ms 46.1 ms 83.6 ms 159 ms 307 ms
GLM-5.3 8.30 ms 8.38 ms 8.68 ms 8.96 ms 9.49 ms 10.5 ms 12.4 ms 16.2 ms 23.7 ms
MiniMax-M3.1 Flash 4.32 ms 4.44 ms 4.71 ms 5.38 ms 6.33 ms 8.62 ms 13.2 ms 22.5 ms 41.3 ms
DeepSeek V4 Pro 6.57 ms 6.57 ms 6.60 ms 6.92 ms 7.24 ms 7.87 ms 9.02 ms 11.2 ms 15.8 ms
DeepSeek V4.1 Flash 3.17 ms 3.19 ms 3.26 ms 3.36 ms 3.52 ms 3.83 ms 4.45 ms 5.67 ms 8.08 ms
Kimi K3 18.7 ms 19.0 ms 19.9 ms 21.7 ms 25.5 ms 31.1 ms 48.8 ms 79.4 ms 141 ms
Kimi K3.1 Pro 20.8 ms 20.9 ms 21.2 ms 21.5 ms 22.1 ms 23.3 ms 25.4 ms 29.8 ms 38.4 ms
GLM-5.3-Flash 6.01 ms 6.01 ms 6.04 ms 6.07 ms 6.18 ms 6.35 ms 6.61 ms 7.10 ms 8.16 ms
Qwen3.8 Flash Next 4.51 ms 4.52 ms 4.52 ms 4.56 ms 4.63 ms 4.78 ms 5.09 ms 5.56 ms 6.54 ms
Qwen3.8 27B 5.27 ms 5.87 ms 7.05 ms 9.40 ms 14.1 ms 23.5 ms 42.3 ms 79.9 ms 155 ms

All of this figure's data (every setting and operation): data/decode_time_grid.md

Decode attention mostly moves data, so its time should follow the bytes that a step reads. The figure below shows how fast each model's attention kernels move these bytes, as their achieved memory bandwidth: the bytes that a decode step reads, divided by the time of its attention kernels. Only the baseline and Qwen3.8 27B come close to the GPU's limit. At 128K tokens, they reach about 7 TB/s, near the B200's peak of 8 TB/s. The new designs do not turn all of their byte savings into time. DeepSeek V4.1 Flash reads 250 times fewer bytes than the baseline but is only 44 times faster, because its sparse attention takes about 1 ms per step at every context length. Inkling's time does not follow its bytes either, because its kernels reach only about 2.3 TB/s at 128K tokens.

Figure data as text: Achieved memory bandwidth of decode attention #

Per model by context length (Batch 32)

Model 4K 8K 16K 32K 64K 128K 256K 512K 1M
Qwen3-Coder-480B-A35B 5.81 TB/s 6.37 TB/s 6.68 TB/s 6.83 TB/s 6.99 TB/s 7.02 TB/s 7.03 TB/s not supported beyond 256K not supported beyond 256K
MiMo V2.6 Flash 1.93 TB/s 2.61 TB/s 3.11 TB/s 3.47 TB/s 3.70 TB/s 3.83 TB/s 3.92 TB/s 3.97 TB/s 4.10 TB/s
Inkling 1.43 TB/s 1.62 TB/s 1.82 TB/s 1.99 TB/s 2.18 TB/s 2.33 TB/s 2.42 TB/s 2.47 TB/s 2.51 TB/s
GLM-5.3 2.53 TB/s 2.65 TB/s 2.61 TB/s 2.97 TB/s 3.50 TB/s 4.19 TB/s 4.87 TB/s 5.38 TB/s 5.73 TB/s
MiniMax-M3.1 Flash 2.34 TB/s 2.66 TB/s 3.18 TB/s 3.77 TB/s 4.84 TB/s 5.63 TB/s 6.20 TB/s 6.54 TB/s 6.69 TB/s
DeepSeek V4 Pro 1.50 TB/s 1.61 TB/s 1.79 TB/s 1.45 TB/s 1.74 TB/s 2.16 TB/s 2.69 TB/s 3.25 TB/s 3.63 TB/s
DeepSeek V4.1 Flash 0.31 TB/s 0.35 TB/s 0.43 TB/s 0.58 TB/s 0.84 TB/s 1.23 TB/s 1.71 TB/s 2.19 TB/s 2.58 TB/s
Kimi K3 5.53 TB/s 5.56 TB/s 5.36 TB/s 5.10 TB/s 4.71 TB/s 4.84 TB/s 4.06 TB/s 3.94 TB/s 3.86 TB/s
Kimi K3.1 Pro 4.39 TB/s 4.38 TB/s 4.31 TB/s 4.33 TB/s 4.42 TB/s 4.59 TB/s 4.89 TB/s 5.22 TB/s 5.55 TB/s
GLM-5.3-Flash 4.01 TB/s 4.04 TB/s 4.03 TB/s 4.06 TB/s 4.03 TB/s 4.06 TB/s 4.21 TB/s 4.48 TB/s 4.78 TB/s
Qwen3.8 Flash Next 4.01 TB/s 4.03 TB/s 4.06 TB/s 4.09 TB/s 4.15 TB/s 4.22 TB/s 4.34 TB/s 4.71 TB/s 5.16 TB/s
Qwen3.8 27B 5.81 TB/s 6.08 TB/s 6.41 TB/s 6.74 TB/s 6.97 TB/s 7.13 TB/s 7.21 TB/s 7.26 TB/s 7.28 TB/s

Color scale: 0 TB/s to 8 TB/s, the B200's peak.

All of this figure's data (every setting and operation): data/decode_bandwidth_grid.md

The rest of the attention module also takes time, mostly in the QKV and output projections. This cost does not grow with the context, because each step reads the same projection weights. For the baseline at 128K tokens, it is only 4% of the step. The new models, however, have cut their attention kernels so far that the projections now take a large share: 67% in GLM-5.3, 60% in Kimi K3.1 Pro and 55% in DeepSeek V4.1 Flash. In these three models, even an infinitely fast attention kernel would shorten the step by less than half. With a single request, the projections and other small kernels take more than half of the step in every new model.

Next, we look at prefill: how much attention work one prompt needs, and how long it takes. The new models need far less attention work than the baseline at long prompts, but several of them save much less time, because their kernels run well below the GPU's peak.

Attention FLOPs per prompt

We first count the attention FLOPs to prefill one prompt, adding up every layer of the model. The colors of each bar show how much of the work each kind of attention does. A sparse layer attends to a fixed number of selected tokens, so its attention grows only linearly with the prompt. Its indexer, however, still scores every earlier token, so at long prompts the indexer does most of the work.

Figure data as text: Attention FLOPs to prefill one prompt #

whole model, one copy · score, weighted sum, indexer scoring and linear-layer chunks; projections, softmax and top-k not counted

Total per model by prompt length

Model 4K 8K 16K 32K 64K 128K 256K 512K 1M
Qwen3-Coder-480B-A35B 25.6 TFLOP 102 TFLOP 409 TFLOP 1.64 PFLOP 6.54 PFLOP 26.2 PFLOP 105 PFLOP not supported beyond 256K not supported beyond 256K
MiMo V2.6 Flash 3.92 TFLOP 14.0 TFLOP 52.8 TFLOP 205 TFLOP 805 TFLOP 3.19 PFLOP 12.7 PFLOP 50.8 PFLOP 203 PFLOP
Inkling 6.57 TFLOP 19.4 TFLOP 63.3 TFLOP 224 TFLOP 834 TFLOP 3.22 PFLOP 12.6 PFLOP 50.0 PFLOP 199 PFLOP
GLM-5.3 69.8 TFLOP 165 TFLOP 365 TFLOP 799 TFLOP 1.80 PFLOP 4.37 PFLOP 11.7 PFLOP 35.3 PFLOP 118 PFLOP
MiniMax-M3.1 Flash 12.9 TFLOP 30.9 TFLOP 70.1 TFLOP 161 TFLOP 392 TFLOP 1.05 PFLOP 3.16 PFLOP 10.6 PFLOP 38.0 PFLOP
DeepSeek V4 Pro 26.3 TFLOP 72.3 TFLOP 174 TFLOP 414 TFLOP 1.05 PFLOP 2.91 PFLOP 9.03 PFLOP 30.9 PFLOP 113 PFLOP
DeepSeek V4.1 Flash 6.40 TFLOP 13.6 TFLOP 29.0 TFLOP 63.9 TFLOP 150 TFLOP 389 TFLOP 1.13 PFLOP 3.67 PFLOP 13.0 PFLOP
Kimi K3 16.9 TFLOP 58.5 TFLOP 216 TFLOP 828 TFLOP 3.24 PFLOP 12.8 PFLOP 51.0 PFLOP 203 PFLOP 812 PFLOP
Kimi K3.1 Pro 37.7 TFLOP 89.2 TFLOP 202 TFLOP 468 TFLOP 1.16 PFLOP 3.17 PFLOP 9.73 PFLOP 33.0 PFLOP 120 PFLOP
GLM-5.3-Flash 10.7 TFLOP 24.9 TFLOP 54.3 TFLOP 118 TFLOP 263 TFLOP 625 TFLOP 1.64 PFLOP 4.83 PFLOP 15.9 PFLOP
Qwen3.8 Flash Next 3.06 TFLOP 6.79 TFLOP 14.4 TFLOP 30.3 TFLOP 64.4 TFLOP 143 TFLOP 339 TFLOP 889 TFLOP 2.62 PFLOP
Qwen3.8 27B 4.87 TFLOP 16.3 TFLOP 59.1 TFLOP 224 TFLOP 870 TFLOP 3.43 PFLOP 13.6 PFLOP 54.2 PFLOP 217 PFLOP

Takeaways the figure shows as the slider moves

  • 4K–16K: At short prompts, selecting a fixed number of tokens saves little. At 4K tokens, GLM-5.3, Kimi K3.1 Pro and DeepSeek V4 Pro even need more attention FLOPs than the baseline, and GLM-5.3 still does at 8K.
  • 32K–128K: The baseline's FLOPs grow with the square of the prompt, while the sparse models grow about twice per doubling. At 128K tokens, Qwen3.8 Flash Next needs 183× less than the baseline.
  • 256K–1M: The indexer still scores every earlier token, so its share keeps rising. At 1M tokens, it is 60% to 90% of the attention FLOPs in the sparse models.

All of this figure's data (every setting and operation): data/prefill_flops.md

Figure data as text: Attention FLOPs to prefill one prompt as the prompt grows #

Per model by prompt length

Model 4K 8K 16K 32K 64K 128K 256K 512K 1M
Qwen3-Coder-480B-A35B 25.6 TFLOP 102 TFLOP 409 TFLOP 1.64 PFLOP 6.54 PFLOP 26.2 PFLOP 105 PFLOP not supported beyond 256K not supported beyond 256K
MiMo V2.6 Flash 3.92 TFLOP 14.0 TFLOP 52.8 TFLOP 205 TFLOP 805 TFLOP 3.19 PFLOP 12.7 PFLOP 50.8 PFLOP 203 PFLOP
Inkling 6.57 TFLOP 19.4 TFLOP 63.3 TFLOP 224 TFLOP 834 TFLOP 3.22 PFLOP 12.6 PFLOP 50.0 PFLOP 199 PFLOP
GLM-5.3 69.8 TFLOP 165 TFLOP 365 TFLOP 799 TFLOP 1.80 PFLOP 4.37 PFLOP 11.7 PFLOP 35.3 PFLOP 118 PFLOP
MiniMax-M3.1 Flash 12.9 TFLOP 30.9 TFLOP 70.1 TFLOP 161 TFLOP 392 TFLOP 1.05 PFLOP 3.16 PFLOP 10.6 PFLOP 38.0 PFLOP
DeepSeek V4 Pro 26.3 TFLOP 72.3 TFLOP 174 TFLOP 414 TFLOP 1.05 PFLOP 2.91 PFLOP 9.03 PFLOP 30.9 PFLOP 113 PFLOP
DeepSeek V4.1 Flash 6.40 TFLOP 13.6 TFLOP 29.0 TFLOP 63.9 TFLOP 150 TFLOP 389 TFLOP 1.13 PFLOP 3.67 PFLOP 13.0 PFLOP
Kimi K3 16.9 TFLOP 58.5 TFLOP 216 TFLOP 828 TFLOP 3.24 PFLOP 12.8 PFLOP 51.0 PFLOP 203 PFLOP 812 PFLOP
Kimi K3.1 Pro 37.7 TFLOP 89.2 TFLOP 202 TFLOP 468 TFLOP 1.16 PFLOP 3.17 PFLOP 9.73 PFLOP 33.0 PFLOP 120 PFLOP
GLM-5.3-Flash 10.7 TFLOP 24.9 TFLOP 54.3 TFLOP 118 TFLOP 263 TFLOP 625 TFLOP 1.64 PFLOP 4.83 PFLOP 15.9 PFLOP
Qwen3.8 Flash Next 3.06 TFLOP 6.79 TFLOP 14.4 TFLOP 30.3 TFLOP 64.4 TFLOP 143 TFLOP 339 TFLOP 889 TFLOP 2.62 PFLOP
Qwen3.8 27B 4.87 TFLOP 16.3 TFLOP 59.1 TFLOP 224 TFLOP 870 TFLOP 3.43 PFLOP 13.6 PFLOP 54.2 PFLOP 217 PFLOP

All of this figure's data (every setting and operation): data/prefill_flops_grid.md

Attention time per prompt

We then measure the attention time of the same prompts on one B200. Each prompt is prefilled in chunks of 8K tokens. Each bar adds up the time of all its chunks. As in decode, the colors show each kind of attention, and gray segments add the rest of each attention module's work.

Figure data as text: Measured attention time to prefill one prompt #

ms to prefill the prompt in 8K chunks, whole model · B200, DP attention (all heads on each GPU). Hover a bar for the backend.

Total per model by prompt length (With other ops)

Model 4K 8K 16K 32K 64K 128K 256K 512K 1M
Qwen3-Coder-480B-A35B 70.1 ms 151 ms 425 ms 1,329 ms 4,570 ms 16,826 ms 64,641 ms not supported beyond 256K not supported beyond 256K
MiMo V2.6 Flash 29.6 ms 62.2 ms 138 ms 330 ms 873 ms 2,604 ms 8,594 ms 30,573 ms 115,938 ms
Inkling 82.3 ms 167 ms 355 ms 788 ms 1,893 ms 5,091 ms 15,378 ms 51,402 ms 184,970 ms
GLM-5.3 148 ms 347 ms 704 ms 1,448 ms 3,064 ms 6,736 ms 15,716 ms 40,148 ms 116,001 ms
MiniMax-M3.1 Flash 55.0 ms 118 ms 249 ms 518 ms 1,091 ms 2,371 ms 5,525 ms 14,344 ms 41,901 ms
DeepSeek V4 Pro 126 ms 271 ms 544 ms 1,095 ms 2,247 ms 4,966 ms 12,167 ms 34,113 ms 109,076 ms
DeepSeek V4.1 Flash 17.2 ms 33.4 ms 67.4 ms 138 ms 289 ms 634 ms 1,497 ms 4,010 ms 12,629 ms
Kimi K3 296 ms 596 ms 1,259 ms 2,778 ms 6,578 ms 17,277 ms 51,040 ms 168,021 ms 599,805 ms
Kimi K3.1 Pro 327 ms 703 ms 1,418 ms 2,880 ms 5,941 ms 12,565 ms 27,690 ms 65,380 ms 171,781 ms
GLM-5.3-Flash 86.1 ms 173 ms 348 ms 699 ms 1,414 ms 2,896 ms 6,091 ms 13,409 ms 32,604 ms
Qwen3.8 Flash Next 54.6 ms 113 ms 233 ms 474 ms 967 ms 1,992 ms 4,236 ms 9,210 ms 20,542 ms
Qwen3.8 27B 43.8 ms 89.1 ms 192 ms 437 ms 1,085 ms 2,996 ms 9,514 ms 33,426 ms 123,175 ms

Backend per model

Model Backend
Qwen3-Coder-480B-A35B FlashInfer trtllm-gen paged prefill/decode, FP8 Q/KV
MiMo V2.6 Flash vLLM FlashAttention 4, FP8 KV, with flash-attention PR #2918 (unpatched FA4 FP8 split-KV decode is wrong)
Inkling per cell the faster of vLLM Inkling fork tml_fa4 with a BF16 cache and SGLang FA4 with MXFP8 Q/K/V (block-scaled QK, in-kernel V dequant), both with the sheared relative bias; Triton fused_qkvr_prep (short conv + q/k norm + cache write)
GLM-5.3 FlashInfer trtllm-gen FP8 sparse MLA, DeepGEMM FP8 indexer, vLLM top-k (cooperative <= 64 rows)
MiniMax-M3.1 Flash vLLM MiniMax-M3 MSA with FP8 K/V and index keys; decode indexer: CuTe DSL index scoring + FlashInfer top-k; decode attention: Triton sparse GQA below 16 requests, CUTLASS fmha_sm100 from 16; prefill indexer: fmha_sm100 score-only (score + block max pool) + FlashInfer top-k; prefill attention: CuTe DSL KV-outer sparse attention + combine
DeepSeek V4 Pro vLLM FlashMLA sparse MLA on the fp8_ds_mla cache, all 128 heads: BF16 prefill (flash_mla_sparse_fwd), FP8 decode under CUDA graphs (Sm100 Head128 split-KV); CSA (C4) layers attend the 128-token window + top-1024 compressed entries, HCA (C128) layers the window + every compressed entry, one call per layer; DeepGEMM FP8 indexer logits, vLLM top-k (cooperative <= 64 rows, persistent above)
DeepSeek V4.1 Flash FlashMLA (compressed sparse MLA; CuTe DSL swap-AB decode for steps of <= 16 requests), DeepGEMM FP8 indexer, vLLM top-k; prefill with decoder bounded replay, as vLLM's V4.1 fork runs it by default
Kimi K3 FlashInfer cute-dsl MLA decode (FP8 q/KV, all 96 heads, no padding); TRT-LLM-gen ragged prefill (vLLM TRTLLM_RAGGED, FP8 Q/K/V); FlashKDA prefill and fused CUDA KDA decode as Kimi K3.1 Pro
Kimi K3.1 Pro vLLM FlashMLA sparse MLA on the fp8_ds_mla cache: FP8 decode (BF16 q padded to 128 heads); prefill upconverts the context to BF16 (BF16 sparse attention at 128 heads); DeepGEMM FP8 indexer, vLLM top-k (cooperative <= 64 rows); FlashKDA prefill, vLLM fused CUDA KDA decode (fused_kda_decode)
GLM-5.3-Flash FlashInfer trtllm-gen FP8 sparse MLA, DeepGEMM FP8 indexer, vLLM top-k (cooperative <= 64 rows); FlashKDA prefill, FLA Triton KDA decode
Qwen3.8 Flash Next vLLM Triton sparse GQA, vLLM top-k (cooperative <= 64 rows); FlashInfer / Triton GDN
Qwen3.8 27B FlashInfer trtllm-gen paged prefill/decode, FP8 Q/KV (FA4 has no FP8-KV path at head dim 256); FlashInfer CUTLASS GDN chunked prefill, vLLM Triton GDN decode

Takeaways the figure shows as the slider moves

  • 4K–16K: At short prompts, sparse attention saves little. At 8K tokens, the baseline's dense attention is still faster than the attention of GLM-5.3, DeepSeek V4 Pro, Kimi K3 and Kimi K3.1 Pro.
  • 32K–128K: The baseline's attention time grows with the square of the prompt, and from 32K tokens on its attention is slower than every new model's. At 128K tokens, its attention takes 15.6 s, while the new models' take 1.6× to 46× less.
  • 256K–1M: At long prompts, the indexer takes a growing share of the time. At 1M tokens, it is most of the time in every sparse model except Qwen3.8 Flash Next.

All of this figure's data (every setting and operation): data/prefill_time.md

Figure data as text: Measured attention time to prefill one prompt as the prompt grows #

Per model by prompt length (With other ops)

Model 4K 8K 16K 32K 64K 128K 256K 512K 1M
Qwen3-Coder-480B-A35B 70.1 ms 151 ms 425 ms 1.33 s 4.57 s 16.8 s 64.6 s not supported beyond 256K not supported beyond 256K
MiMo V2.6 Flash 29.6 ms 62.2 ms 138 ms 330 ms 873 ms 2.60 s 8.59 s 30.6 s 116 s
Inkling 82.3 ms 167 ms 355 ms 788 ms 1.89 s 5.09 s 15.4 s 51.4 s 185 s
GLM-5.3 148 ms 347 ms 704 ms 1.45 s 3.06 s 6.74 s 15.7 s 40.1 s 116 s
MiniMax-M3.1 Flash 55.0 ms 118 ms 249 ms 518 ms 1.09 s 2.37 s 5.53 s 14.3 s 41.9 s
DeepSeek V4 Pro 126 ms 271 ms 544 ms 1.09 s 2.25 s 4.97 s 12.2 s 34.1 s 109 s
DeepSeek V4.1 Flash 17.2 ms 33.4 ms 67.4 ms 138 ms 289 ms 634 ms 1.50 s 4.01 s 12.6 s
Kimi K3 296 ms 596 ms 1.26 s 2.78 s 6.58 s 17.3 s 51.0 s 168 s 600 s
Kimi K3.1 Pro 327 ms 703 ms 1.42 s 2.88 s 5.94 s 12.6 s 27.7 s 65.4 s 172 s
GLM-5.3-Flash 86.1 ms 173 ms 348 ms 699 ms 1.41 s 2.90 s 6.09 s 13.4 s 32.6 s
Qwen3.8 Flash Next 54.6 ms 113 ms 233 ms 474 ms 967 ms 1.99 s 4.24 s 9.21 s 20.5 s
Qwen3.8 27B 43.8 ms 89.1 ms 192 ms 437 ms 1.09 s 3.00 s 9.51 s 33.4 s 123 s

All of this figure's data (every setting and operation): data/prefill_time_grid.md

To compare time with work, we use the achieved compute throughput: the attention FLOPs of a prompt divided by the time of its attention kernels. The figure below shows it for every model and prompt length.

Figure data as text: Achieved compute throughput of prefill attention #

Per model by prompt length

Model 4K 8K 16K 32K 64K 128K 256K 512K 1M
Qwen3-Coder-480B-A35B 0.81 PFLOP/s 1.37 PFLOP/s 1.51 PFLOP/s 1.60 PFLOP/s 1.65 PFLOP/s 1.68 PFLOP/s 1.68 PFLOP/s not supported beyond 256K not supported beyond 256K
MiMo V2.6 Flash 0.73 PFLOP/s 1.02 PFLOP/s 1.27 PFLOP/s 1.50 PFLOP/s 1.66 PFLOP/s 1.75 PFLOP/s 1.81 PFLOP/s 1.85 PFLOP/s 1.85 PFLOP/s
Inkling 0.29 PFLOP/s 0.39 PFLOP/s 0.53 PFLOP/s 0.70 PFLOP/s 0.87 PFLOP/s 1.00 PFLOP/s 1.09 PFLOP/s 1.14 PFLOP/s 1.17 PFLOP/s
GLM-5.3 2.07 PFLOP/s 1.39 PFLOP/s 1.47 PFLOP/s 1.49 PFLOP/s 1.46 PFLOP/s 1.42 PFLOP/s 1.39 PFLOP/s 1.38 PFLOP/s 1.36 PFLOP/s
MiniMax-M3.1 Flash 0.41 PFLOP/s 0.45 PFLOP/s 0.47 PFLOP/s 0.50 PFLOP/s 0.56 PFLOP/s 0.66 PFLOP/s 0.80 PFLOP/s 0.94 PFLOP/s 1.07 PFLOP/s
DeepSeek V4 Pro 0.77 PFLOP/s 0.83 PFLOP/s 0.99 PFLOP/s 1.16 PFLOP/s 1.23 PFLOP/s 1.26 PFLOP/s 1.30 PFLOP/s 1.30 PFLOP/s 1.28 PFLOP/s
DeepSeek V4.1 Flash 0.87 PFLOP/s 0.91 PFLOP/s 0.95 PFLOP/s 1.00 PFLOP/s 1.07 PFLOP/s 1.15 PFLOP/s 1.25 PFLOP/s 1.30 PFLOP/s 1.27 PFLOP/s
Kimi K3 0.28 PFLOP/s 0.44 PFLOP/s 0.66 PFLOP/s 0.91 PFLOP/s 1.16 PFLOP/s 1.35 PFLOP/s 1.46 PFLOP/s 1.53 PFLOP/s 1.57 PFLOP/s
Kimi K3.1 Pro 0.44 PFLOP/s 0.39 PFLOP/s 0.43 PFLOP/s 0.48 PFLOP/s 0.54 PFLOP/s 0.64 PFLOP/s 0.78 PFLOP/s 0.94 PFLOP/s 1.08 PFLOP/s
GLM-5.3-Flash 0.39 PFLOP/s 0.44 PFLOP/s 0.47 PFLOP/s 0.51 PFLOP/s 0.54 PFLOP/s 0.61 PFLOP/s 0.69 PFLOP/s 0.81 PFLOP/s 0.90 PFLOP/s
Qwen3.8 Flash Next 0.11 PFLOP/s 0.11 PFLOP/s 0.11 PFLOP/s 0.11 PFLOP/s 0.12 PFLOP/s 0.12 PFLOP/s 0.13 PFLOP/s 0.15 PFLOP/s 0.19 PFLOP/s
Qwen3.8 27B 0.54 PFLOP/s 0.81 PFLOP/s 1.10 PFLOP/s 1.39 PFLOP/s 1.63 PFLOP/s 1.81 PFLOP/s 1.86 PFLOP/s 1.87 PFLOP/s 1.89 PFLOP/s

Color scale: 0 PFLOP/s to 4.5 PFLOP/s, the B200's FP8 peak.

All of this figure's data (every setting and operation): data/prefill_throughput_grid.md

At 128K tokens, the baseline, MiMo V2.6 Flash, Qwen3.8 27B, Kimi K3, GLM-5.3 and DeepSeek V4 Pro all reach 1.3 to 1.8 PFLOP/s on one B200, so their prefill time roughly follows their FLOPs. GLM-5.3 spends more than a third of its attention time on the indexer and the top-k selection. At 1M tokens, these two take most of the attention time in every model with an indexer, except Qwen3.8 Flash Next.

The other new models run their attention more slowly, so their time falls by much less than their FLOPs. Qwen3.8 Flash Next needs 183 times fewer FLOPs than the baseline but is only 13 times faster. Most of that time goes to its sparse attention, which reaches only about 90 TFLOP/s. Kimi K3.1 Pro's attention reaches only about 640 TFLOP/s, because vLLM runs its sparse MLA prefill in BF16 and pads its 96 heads to 128. Inkling needs about as many attention FLOPs as MiMo V2.6 Flash, but its attention takes 1.8 times as long and reaches about 1.0 PFLOP/s. Both models run FlashAttention 4, but Inkling's version also adds a learned position bias to every score, which may account for part of the gap.

Conclusion and Get Started #

Attention in open models is no longer one design. At 128K tokens, the eleven new models keep 1.8× to 139× less attention memory than a GQA model of similar size, and they read and compute far less in decode and prefill. The time saved is smaller than the work saved, because the new designs add work, such as indexers, and some of their kernels run far below the GPU's peak. Inference analysis and optimization should start from these models.

The kernel times in this post come from ServingStudio Sim's measured kernel profiles. ServingStudio Sim also models whole serving steps, beyond attention. On the Models page, you can explore models such as Inkling, DeepSeek V4 Flash, GLM-5.2 and Qwen3-235B. For each one, the page shows its attention layout and cache parts, the cost tree that builds one step's time from measured kernels, and a live prediction for a batch of your own. The simulator is on GitHub.

── more in #ai-infrastructure 4 stories · sorted by recency
── more on @servingstudio 3 stories trending now
sponsored brought to you by zahid.host 4,200+ EU-deployed projects
reading about agents? ship yours in a single git push.

Run your AI side-project on zahid.host

EU-based hosting, git-push deploys, automatic HTTPS, no cold starts. Free tier with a custom domain — perfect for shipping the agent you just read about.

$git push zahid main
→ Live at https://your-agent.zahid.host ✓
Get free account → Pricing
from €0/mo · no card required
LIVE [news/beyond-gqa-character…] indexed:0 read:67min 2026-11-20 · —