Evolution of Attention over the Years

A technical survey traces five rewrites of transformer attention since 2017, framing every variant as either storing less or reading less. It details how multi-query and grouped-query attention shrink the KV cache from 2NHdₕ to 2NGdₕ per layer by sharing keys and values across query heads, and notes that deployment cost is driven by memory bandwidth rather than the quadratic QKᵀ term. The piece distinguishes deleting information from keeping it and choosing not to look, and observes that labs remain split, with Qwen3.5 and Kimi K3 mixing while DeepSeek and GLM skip.

Attention is the only block in a transformer whose cost grows with the input. Every other block processes one token at a time, so a token near the end costs the same as one at the start. Attention has to compare each new token against all the tokens before it, so the work grows as the input gets longer. Memory is the harder half of that. The model keeps every token it has read, and reads the whole collection again for each new token it produces. At a million tokens of context, writing one word means reading a million entries first. The arithmetic finishes quickly. The memory access does not. So attention kept changing while the rest of the architecture did not. Five rewrites since 2017, and only two moves available: store less, or read less. <sub Figure 1. The five cuts, in order. The first four make the model store less, and each deletes something. The last makes it read less, and deletes nothing.</sub Each is faster, and each breaks something different. One forgets old tokens. One mixes them together. One keeps them all and never reads them. The labs still disagree. Qwen3.5 and Kimi K3 mix, DeepSeek and GLM skip, MiniMax mixed in two models and went back. This article takes the five in order, asking what each saves and what it costs. The distinction that matters most: deleting information is not the same as keeping it and choosing not to look. <sub Symbols used in equations below:</sub <sub N tokens, H heads, d for the model dimension and dₕ for one head. A single token is a column vector and projections act on the left, q t = W Qh t; stacked over the whole sequence this is Q = XW Q^ᵀ, with one token per row. Softmax is applied to each row separately.</sub Multi-head attention MHA : the starting point Full attention deletes nothing. Every query reads every key. For one head, the output at position t is a weighted sum of all earlier values: Written for the whole sequence at once, the weights collect into one matrix: Every attention mechanism in this article computes O = AV for some mixing matrix A, and they differ only in how A is built and which of its entries may be nonzero. M is the causal mask, which stops a token from reading anything that comes after it: <sub Figure 2. The causal mask in three steps. Raw scores exist for every pair of tokens. Adding M leaves the past untouched and sends the future to minus infinity. After the softmax, the future weights are exactly zero.</sub M has no parameters and is the same in every head and every layer. It is also the slot that later mechanisms change: a sliding window narrows it to a band, and sparse attention fills it from a learned score. Two costs come out of this. The QK^ᵀ term is N × N, which is the familiar quadratic. The one that matters more in deployment is the cache: The cache grows with sequence length and is read in full at every decode step, which makes generation limited by memory bandwidth rather than arithmetic. Almost everything below targets the cache, not the quadratic. Multi-query and grouped-query attention MQA, GQA : sharing keys and values The problem. The cache is 2NHdₕ per layer. The H in this expression is the expensive part. A model with 64 heads keeps 64 key vectors and 64 value vectors for every token it has read, and reads all of them at every step. But heads differ mainly in their queries. There is no strong reason for the keys to differ as well. The fix. Keep a separate query projection for each head, but let several heads share one set of keys and values. Divide the H query heads into G groups, and store one key and one value per group instead of one per head. Let g h be the group of head h: Multi-query attention is the case G = 1, where all heads share one key and one value. Grouped-query attention is the middle case, 1 < G < H. Setting G = H gives back full multi-head attention. The cache falls from 2NHdₕ to 2NGdₕ. In matrix form, only the superscripts change: <sub Figure 3. Eight query heads in all three. Multi-head attention gives each head its own key and value. Grouped-query attention shares one pair between several heads. Multi-query attention shares one pair across all of them. Each step down cuts the cache; each step down also cuts how many different things the heads can look for.</sub What is lost. When a head compares the current token with an earlier one, two matrices decide how. W K⁽ʰ⁾ projects the earlier token down to dₕ dimensions, which fixes what is available to match against. W Q⁽ʰ⁾ projects the current token into that same space, which fixes how those available features are weighted. In full attention every head has its own W K⁽ʰ⁾, so every head defines its own dₕ-dimensional view of the past. Those views can differ. One head can expose the features that identify a verb, another the features that identify a name, another the position. Under grouped-query attention the heads in a group share one W K^ g , so they all see the same view. Each head can still weight that view differently through its own W Q⁽ʰ⁾, but no head can match on a feature the shared projection left out. The number of distinct views of the past drops from H to G. The measured quality cost was small. This is why grouped-query attention became a default rather than a documented approximation. The last row explains the reason. Nothing about the tokens was deleted. Every token still contributes a complete key and a complete value. The only limit is on how many different ways the model can look at them. Multi-head latent attention MLA : one small summary instead of many keys The problem. Grouped-query attention helps, but only to a point. If G is reduced to 1, quality drops, because every head is then restricted to the same part of each token. The cache is still 2NGdₕ, still proportional to the number of tokens, and still read at every step. The fix. DeepSeek-V2 keeps per-head keys and values but does not store them. Each token is compressed once into a short summary vector c s of length r. Each head then generates its key and value from that summary when required: Only c s is stored, so the cache falls from 2NHdₕ to Nr. Why the expansion costs nothing. At first this looks like it only moves the problem. To score a past token, a head needs that token’s key. The key is no longer stored, so it must be rebuilt from the summary. This rebuilding would happen for every token in the history, at every step. The cache is smaller, but the arithmetic is larger. The solution comes from looking at what the score is. It is a single number: the query multiplied with the key. Written out, the rebuilding step appears in the middle: The rebuilding matrix does not have to stay on that side. Moving it to the other side gives the same number: Both sides give the same score, but they do different work. The left side expands every stored summary into a full key. The right side shrinks the query instead, and compares it against the summaries exactly as they are stored. That asymmetry is the whole trick. There is one query per step and there are thousands of past tokens, so converting the query once is far cheaper than converting the entire history. Keys are never rebuilt at all, and values work the same way. The conversion is not even a real cost. W UK⁽ʰ⁾ is fixed after training, so it can be folded into the query weights in advance, and the model produces the shrunken query directly. Stack the summaries into one matrix, with a row per token. Written out for the whole sequence, keys and values no longer appear anywhere in the layer: Everything the layer reads from the past is now C, with one exception. Positions do not absorb. Rotary embeddings rotate each vector by an amount that depends on where it sits. Once the rotations are applied, a rotation that depends on s appears between the query and the up-projection, so the up-projection can no longer be moved onto the query in advance: DeepSeek’s answer is to keep positions out of the compressed path. Each token gets a second, short key k s^R ∈ mathbbR^d R that carries the rotation and is shared by all heads. The content half still absorbs; the positional half is computed directly. The cache therefore holds two things per token, not one: the summary of length r and the positional key of length d R. The true footprint is N r + d R , with d R small next to r. What is lost. Every key and value that a token can produce is built from the same r numbers. The H keys for a token are therefore not H independent vectors. They are H different readings of one short summary. Anything the compression removed is removed for all of them at the same time: The assumption is that a token’s keys and values never required more than r numbers of variation. <sub Figure 4. What each mechanism keeps in the inference cache. MHA stores a key and value for every head. GQA and MQA share them between heads. MLA stores one short summary and rebuilds the rest when needed.</sub The last row is the first entry in this article that is not “all.” Grouped-query attention limited how many ways the model can look at a token. MLA limits what there is to look at. In practice the cost was still small, and two inexpensive reductions in a row created an expectation that the next one would also be inexpensive. One implementation detail. Rotary position embeddings depend on the distance t - s between two tokens, so the rotation cannot be combined with a fixed weight in the way W UK⁽ʰ⁾ was. MLA therefore keeps a small number of separate position-carrying dimensions next to the summary. Sliding-window attention SWA : forgetting distance The problem. MLA reduces the cost per token, but the cache still grows with the number of tokens. If the model must keep something for every token it has read, the total will keep rising. The fix. Stop looking further back than a fixed distance w. Nothing about the attention operation changes. The only difference is which entries the mask allows: The mixing matrix is now a band running down the diagonal instead of a full triangle. The number of entries per row stops growing with the input, and so does the cache: Depth recovers part of the loss. One layer reaches back w tokens, but a second layer reads tokens that already absorbed w tokens of their own. Stacked L deep, information can travel roughly L · w tokens, though it arrives blended rather than exact. What is lost. This loss is different from the two before it. Nothing was compressed. Tokens outside the window leave nothing behind, so no later layer can recover them. This cost was measured directly. MiniMax ran an ablation over hundreds of billions to trillions of continued-pretraining tokens. They varied the ratio of windowed to full layers, the position-embedding settings, the mixing within and between layers, and the use of attention-sink tokens, and they analyzed the retrieval and induction heads that resulted. Every configuration was worse at retrieval, multi-hop reasoning, and in-context learning, and the gap grew beyond 32K context after supervised fine-tuning. Linear attention: one running summary instead of a cache Qwen3.5, Kimi K3, and MiniMax-01 all shipped with linear attention in most of their layers. This is the largest change in this article, and the only one that replaces the attention operation rather than restricting it. The problem. Every mechanism so far still keeps something for each token. MLA reduced the cost to r numbers per token, but r numbers multiplied by one million tokens is still one million times r. As long as the model keeps a separate entry per token, the cache grows with the input and each step becomes slower. The fix. Do not keep a separate entry per token. Keep one running summary of the entire past, and update it as each token arrives. The softmax prevents this. To weight a past token, the model must apply the exponential to the query-key score, and the exponential can only be applied after the two vectors have been multiplied together. Every pair must therefore be handled separately, and there are N pairs at every step. Linear attention removes the exponential. Each query and each key is passed through a fixed function φ first, and the score is their dot product: This appears to be a small change, but it allows the sum to be regrouped. Attention adds every past value, weighted by its score. Without the exponential, the query can be moved outside the sum: The left side walks through the whole history for every query. The right side collects the history into one matrix first, and then multiplies that matrix by the query once. The matrix does not depend on the query, so it can be built as the input arrives and reused at every step. Call this matrix S, and consider what it contains. Each token contributes one term, v sφ k s ^ᵀ. This is a value vector of length d v multiplied with a key vector of length r, which produces a d v × r grid of numbers. The next token produces a grid of the same shape, and the two grids are added together entry by entry. This is the important point. The terms are not placed side by side. They are added on top of each other. Ten tokens and ten million tokens both leave S at size d v × r, because addition does not change the shape of what it adds. The layer therefore performs only two operations. A new token is added into S, and a query is answered by multiplying with S: The layer has become a recurrent network. There is no cache and nothing to re-read. There is only one matrix, updated in place. Compute per token stops growing, and memory stops growing with it. Two details matter in implementation but not for the argument here: the output is divided by a running normalizer, and training splits the sequence into chunks so that the recurrence does not have to run one token at a time. What is lost. Every token was added into the same matrix. After this, there is no way back. It is not possible to look at S and extract what token 400 contributed, because its numbers were added into the same cells as every other token’s. Two costs follow. Neither is something that better training can fix. The matrix runs out of room. S has d v × r cells, and the model keeps adding to them. It can hold about r items separately. Beyond that, a query returns the item it asked for mixed with weak traces of everything else: Two tokens stay separate in S only if their keys point in different directions, and r dimensions contain only r genuinely different directions. Token r+1 must reuse a direction that is already occupied. Nothing was configured incorrectly. There was simply no separate place to put it. The weights cannot be sharp. Softmax can place almost all of its weight on a single token. If the query is scaled up, the gap between the best match and the rest grows without limit, until everything except the winner is reduced to zero: Linear attention has no equivalent control. Its weights are plain dot products, and scaling the query multiplies the normalizer by the same factor, so the weights do not change. However confident the model is, the best match cannot receive much more weight than the second-best. Retrieving one exact fact is the case that needs this gap the most. <sub Figure 5. Softmax attention keeps each token in its own slot, so the cache grows and each token remains exactly itself. Linear attention adds all tokens into one fixed matrix, which never grows and never separates them again.</sub The variants: using a fixed budget better Most work since then has been about using S more carefully. Every version that has shipped in a frontier model is a variant of one idea: control what happens to the old contents when a new token is written. Plain linear attention only adds. Nothing is ever removed, so the state fills with old material and the interference term above grows. DeltaNet erases first. Before writing a new value, it removes whatever was already stored under that key. Updating a fact therefore replaces the old version instead of adding on top of it: Memory becomes something the layer edits rather than something it only adds to. Gated DeltaNet GDN adds forgetting. A scalar α t between 0 and 1 fades the whole state slightly at every step, so old material decays instead of remaining. Qwen3-Next and Qwen3.5 use this. Kimi Delta Attention KDA makes the forgetting selective. Instead of one α t that fades everything equally, it uses a separate rate for each channel, so the state can release one kind of information while keeping another. Kimi Linear and K3 use this. Gated DeltaNet-2 GDN-2 separates two things that GDN combined. In the rules above, a single β t controls both how strongly the old value is cleared and how strongly the new one is written. GDN-2 gives each its own strength, so the layer can overwrite firmly while writing gently, or the reverse. <sub Figure 6. Writing a token is two steps: clear whatever was stored under the key, then add the new value. DeltaNet, GDN and KDA set both strengths with one number. GDN-2 gives each its own.</sub All of these are the same recurrence with a different rule for what survives: <sub Figure 7. Each variant keeps the same write step and changes only what happens to the state already stored.</sub The differences between these are real and they appear in benchmarks. But one thing does not change: S has the same size in every row. These variants use a fixed budget more carefully. None of them makes the budget larger. Sparse attention NSA, DSA : reading less, keeping everything DeepSeek-V3.2 and V4, and Zhipu’s GLM-5, chose this direction instead of linear attention. It is the only change in this article that removes nothing. The problem. Sliding-window attention and linear attention both save memory by removing information. But most of what a query reads contributes very little. A small number of past tokens carry the signal, and the rest are almost irrelevant. The waste is not that the model stores too much. The waste is that it reads all of it at every step in order to find the few tokens that matter. The fix. Keep the whole cache. Before running attention, select the tokens that are worth reading. A small and inexpensive scorer, called the indexer, rates every past token against the current query. The highest k tokens are selected. DeepSeek Sparse Attention DSA , which followed the earlier Native Sparse Attention NSA , uses k = 2048 in DeepSeek-V3.2: The selected tokens become a mask, and attention then runs exactly as before over what remains: Each query now reads k tokens instead of all of them, so the attention cost stops growing with context length. Two things make this practical. First, the scoring is done in low dimensions with an inexpensive nonlinearity. The indexer does still look at every token, but it costs a small fraction of what real attention would cost. Second, the selection is shared. DeepSeek-V3.2 uses the absorbed form of MLA, in which all heads read the same cached C, so one list of selected tokens serves every head instead of each head selecting its own. Three mechanisms in this article are now the same operation with different rules for what to skip: What is lost. Nothing, in the sense that has applied everywhere else in this article. Every token’s entry is still in the cache and is unchanged. The model only did not read it at this step. This makes the failure different in kind. If the indexer selects the correct k tokens, the result matches full attention exactly. If it selects incorrectly, the required token is still present, and a better indexer would have found it. Linear attention is different: a fact that has been mixed into S cannot be recovered by anything. This is why the later work is almost entirely about the selection. Hierarchical indexing, sharing the index between neighbouring layers in GLM-5.2, and top-k and top-p variants all improve the selection and leave attention itself unchanged. The cache still grows, and this is the real price. Sparse attention reduces compute, not memory. But the last two rows explain why this branch and the linear branch are not competitors. One limits what is stored and cannot undo its losses. The other limits what is read and can always be improved. The mixing matrix, side by side Every mechanism above computes O = AV with A ∈ mathbbR^N × N. Placing the six forms next to each other shows what each one gave up. <sub Figure 8. The mixing matrix. Blue is read at this step, grey is in the cache but not read, white is gone. Only sparse attention leaves grey squares, and that is the distinction this article is built on.</sub Five of the six constrain A and keep the softmax. Only linear attention replaces the operation itself. Of the three that constrain which entries are used, only sparse attention chooses those entries from the content at run time. This is why its errors can be corrected and the others cannot. The trades The last row is different in kind from the others. This is why the two branches are complements rather than competitors, and why production systems are likely to use both. Hybrids: nobody ships these alone No frontier model replaces every attention layer. Linear layers are interleaved with ordinary full-attention layers, and the mix is the actual design decision. The reason is division of labour. Linear layers handle the bulk of the sequence cheaply. The few full layers left in the stack do the work that needs exact lookup. Roughly one full layer for every three to six linear ones is enough for recall to hold up, and this is where the 3:1 schedules now in wide use come from. What the evidence shows Perplexity does not detect the loss. Language modeling metrics stay flat across a wide range of linear-to-full ratios while recall does not. Only long-context retrieval tests of the RULER family separate the good ratios from the bad ones Wang et al., 2025 . An evaluation suite that cannot see a loss will ship it. The field ran the experiment and divided. MiniMax shipped linear hybrids in MiniMax-01 and M1, then returned M2 to full attention everywhere, reporting that no efficient variant reliably matched it in production. Their own account blames evaluation rather than architecture: the hybrids matched full attention on MMLU, BBH, MATH and LongBench, and only fell behind at scale on multi-hop reasoning. Reports on M3 point to block-sparse attention rather than a return to linear. There is a lower bound. Ye et al. 2026 prove that on deep sequential composition, where each retrieval step determines the context of the next, a full attention model with L+1 layers solves problems no hybrid with L full-attention layers can match, even given exponentially many linear layers. The result concerns expressivity, not training. So the ratio has a floor, and that floor is set by the reasoning depth the task requires rather than by the memory budget. More linear layers do not move it. Where the design problem stands The ratio is no longer fixed before training. HALO converts pretrained models into hybrids using under 0.01% of the original token budget. FlashMorph treats layer selection as an optimization problem, gating each full layer against a linear branch and then discretizing to a chosen budget. GLM-5 searched for its configuration. Allocation is also getting finer. NAtS-L routes individual tokens, sending short-lived ones to GDN and keeping likely-to-be-retrieved ones in softmax. HydraHead allocates per head. The direction runs from a fixed constant, to a searched configuration, to a decision the model makes at run time. What I take from this Lossy is not one property. What matters is whether a mechanism deletes information or only declines to read it. Linear and sparse attention are often treated as two versions of one idea. They are not, and that difference explains most of the division above. The early cuts were cheap, and that set expectations. GQA and MLA cost almost nothing, which made the later cuts look cheaper than they were. Measurement is the limiting factor. The loss shows up in multi-hop retrieval at long context, which is exactly the shape of agentic work: a model reads a long trajectory it wrote itself and has to recover an exact fact from far earlier. The workloads that create demand for efficient attention are the ones most sensitive to what it removes. The floor is real. Budget full-attention layers as permanent, and size that budget from the reasoning depth the application needs, not from the memory you would like to reclaim. <sub I wrote this with Claude. The argument, the structure, and the editorial judgment are mine; Claude helped with drafting, background research, and the figures. Everything here reflects my own reading of the literature, and any errors are mine.</sub <sub Thanks</sub <sub Aman Chanda</sub <sub for reviewing and giving feedback</sub References - Shazeer, Fast Transformer Decoding: One Write-Head is All You Need 2019 — MQA - Ainslie et al., GQA 2023 - Katharopoulos et al., Transformers are RNNs 2020 — linear attention - Schlag et al., Linear Transformers Are Secretly Fast Weight Programmers 2021 — delta rule - DeepSeek-AI, DeepSeek-V2 arXiv:2405.04434 — MLA - Yang et al., Gated Delta Networks arXiv:2412.06464 , ICLR 2025 - Yuan et al., Native Sparse Attention 2025 - Wang et al., July 2025 — hybrid ratio / recall saturation study - Kimi Team, Kimi Linear: An Expressive, Efficient Attention Architecture arXiv:2510.26692 - MiniMax, Why Did M2 End Up as a Full Attention Model? ; MiniMax-M2 report arXiv:2605.26494 - DeepSeek-AI, DeepSeek-V3.2 arXiv:2512.02556 — DSA - Ye et al., A Provable Expressiveness Hierarchy in Hybrid Linear-Full Attention arXiv:2602.01763 - Chen et al., Hybrid Linear Attention Done Right arXiv:2601.22156 — HALO / HypeNet - FlashMorph arXiv:2606.30562 ; HydraHead arXiv:2606.20097 ; NAtS-L arXiv:2602.03681