{"slug": "let-s-stop-calling-everything-linear-attention", "title": "Let's Stop Calling Everything \"Linear Attention\"", "summary": "A new critique argues that the term 'linear attention' is overused and ambiguous, covering at least six different resource claims that range from constant computation per token to merely reduced KV cache size. The piece, published on a technical blog, proposes that systems analysis should separate attention mechanisms by memory capacity, memory-load traffic, and computation, and warns that papers often imply all three are linear when only one is proven.", "body_md": "# Let’s Stop Calling Everything “Linear Attention”\n\n*If your “constant-time” attention still scans a million tokens, we need to talk.*\n\n“Linear attention” is becoming the **gluten-free of AI architecture terminology**: everyone puts it on the label, nobody agrees what it means, and you still have to inspect the ingredients.\n\nDepending on the paper, *linear* may mean:\n\n- linear computation during prompt prefill;\n- constant computation per generated token;\n- a constant-size recurrent state;\n- a smaller—but still linearly growing—KV cache;\n- sparse attention over a fixed number of selected tokens; or\n- “most of our layers are linear, except for the ones that aren’t.”\n\nThese are not minor variations of the same claim. They describe **different resource requirements** and can produce radically different behavior on real hardware.\n\nRecent surveys distinguish hardware-efficient, compact, sparse, and algebraically linear attention. That taxonomy is useful, but for systems analysis we need an even cleaner separation: **capacity, traffic, and computation**.\n\nBefore declaring another attention mechanism “linear,” let us ask three questions:\n\n**How much memory does it occupy?****How much data does it load?****How much computation does it perform?**\n\nIf a paper answers only one of these and lets the adjective *linear* imply the other two, hide your asymptotic wallet.\n\n## The Three Bills Attention Sends You\n\nLet *L* be the current sequence length. For simplicity, assume dimensions, head counts, window sizes, state sizes, and selection counts remain fixed as *L* grows.\n\n### 1. Occupied Memory Capacity: What Must Stay Resident?\n\nAfter processing *L* tokens, how much persistent state must the model retain to generate the next one?\n\nConventional attention keeps a key and value representation for every previous token: `M`\n\n. Using grouped-query attention, multi-query attention, quantization, or Multi-Head Latent Attention can make each entry much smaller. This is extremely valuable. But if the architecture still keeps one entry per token, occupied capacity remains KV = O(L)**O(L)**.\n\nA recurrent linear-attention or state-space layer instead folds the history into a fixed-size state: `M`\n\n. That state might be a rather chunky matrix. “Constant” does not mean “tiny”; it means that token number 1,000,001 does not require another cache entry.state = O(1) with respect to L\n\nA KV cache is a warehouse that grows with every token. A recurrent state is a fixed-size suitcase. The suitcase is wonderfully portable—but eventually you must decide what to forget, overwrite, or compress.\n\n### 2. Memory-Load Traffic: What Must Be Dragged Through the Machine?\n\n**Occupied memory is not bandwidth.**\n\nA system can retain a huge history but touch only a few entries for each query. Conversely, a cache may fit comfortably in high-bandwidth memory while the model rereads the entire thing for every generated token.\n\nThe relevant quantity is: `T`\n\n. The corresponding bandwidth floor is approximately: mem(L) = bytes transferred per decode step`t`\n\n. Dense attention usually stores mem >= Tmem / Beffective**O(L)** KV state and reads **O(L)** KV state for every token. The cache is not merely taking up space; it is demanding a complete guided tour on every decode step.\n\nFlashAttention showed why data movement deserves its own axis. It does not change exact attention’s arithmetic asymptotics. Instead, it tiles and fuses the computation so that intermediate results are kept out of high-bandwidth memory wherever possible, greatly reducing IO and temporary materialization.\n\nSame mathematical attention. Same asymptotic flops. Much less pointless furniture moving.\n\nA more interesting design point is data-driven sparse attention, where the system retains an O(L) cache but does not load every full KV entry for every decode step. Instead, a lightweight, query-dependent selector identifies a small set of relevant positions, and only those k KV entries are fetched for the expensive attention operation. If selection can be performed using a compact index—or through a genuinely sublinear retrieval structure—the main KV traffic falls from O(L) to O(k), even though occupied cache capacity remains O(L). In other words, the model keeps the whole library but carries only a few books to the desk. This creates a useful combination that recurrent methods do not offer: unbounded token-addressable memory capacity with potentially constant full-KV traffic per decode step.\n\nDeepSeek Sparse Attention is an instructive—but not yet fully sublinear—example. Its lightning indexer scans compact, low-precision index representations for all preceding tokens, selects the top-k positions, and loads the larger MLA cache entries only for those positions. DSA therefore avoids loading all full KV entries, reducing expensive KV traffic to O(k); however, it still loads or processes O(L) compact index data to discover them. Its total selection traffic consequently remains linear in L, although with a much smaller coefficient than dense MLA. Future hierarchical, hashed, clustered, or learned indexes could push this toward sublinear retrieval—producing architectures with O(L) occupied capacity but sublinear, or even approximately constant, memory traffic and attention computation per decode step.\n\n### 3. Computational Complexity: How Much Arithmetic Happens?\n\nFor dense causal attention:\n\n- prompt prefill costs\n**O(L²)**; - one decode step costs\n**O(L)**; and - generating\n*G*additional tokens costs approximately**O(GL + G²)** for attention.\n\nThese quantities must be reported separately. A mechanism can reduce expensive attention computation without reducing persistent memory. It can reduce memory traffic without changing flop complexity. It can also have linear total prefill\n\ncomplexity while using constant computation per decode step.\n\nCalling all of these “linear” is mathematically legal in roughly the same way that calling a tiger “a cat” is zoologically legal.\n\n## The Attention Complexity Zoo\n\nThe following table summarizes the sequence-length scaling of prominent mechanism families. All **O(1)** entries are with respect to *L*, assuming fixed dimensions, state sizes, windows, ranks, or selection counts.\n\n| Mechanism | Occupied state | Decode traffic | Decode compute | Prefill compute |\n|---|---|---|---|---|\n| Dense MHA | O(L) | O(L) | O(L) | O(L²) |\n| FlashAttention | O(L) KV; smaller working memory | O(L), better IO constants | O(L) | O(L²) |\n| MQA/GQA | O(L), smaller coefficient | O(L), smaller coefficient | O(L) | O(L²) |\n| MLA | O(L), compressed | O(L), compressed | O(L) | O(L²) |\n| Sliding window with eviction | O(1) | O(1) | O(1) | O(L) |\n| Sparse access, all history retained | O(L) | O(1), if locations are known | O(1) | O(L) |\n| DSA-style exhaustive index plus top-k | O(L) | O(L) scan + O(k) retrieval | O(L) indexing + O(k) attention | Potentially O(L²) indexing |\n| Performer/kernel linear attention | O(1) | O(1) | O(1) | O(L) |\n| GLA/DeltaNet/Gated DeltaNet/KDA layer | O(1) | O(1) | O(1) | O(L) |\n| Mamba/SSM layer | O(1) | O(1) | O(1) | O(L) |\n| Fixed-ratio recurrent/dense hybrid | O(L) | O(L) | O(L) | O(L²) |\n\nThe table’s main message is not that one row wins. It is that **the columns can move independently**.\n\n## Dense Attention: Expensive, Honest, and Easy to Classify\n\nDense attention is refreshingly straightforward.\n\nIt retains one KV entry per previous token: `capacity = O(L)`\n\n. It reads and interacts with those entries for each new token: `decode traffic = O(L) decode compute = O(L)`\n\n. During prefill, every prompt token can interact with every earlier token: `prefill compute = O(L²)`\n\n. One frequent source of confusion is the statement that attention requires “quadratic memory.” Naive prefill or training implementations may materialize an *L × L* score matrix, which is indeed quadratic. Persistent inference KV capacity, however, is linear.\n\nFlashAttention avoids materializing that quadratic intermediate in HBM while still performing exact dense attention. It reduces working memory and IO—not the quadratic number of prefill interactions.\n\nFlashAttention is faster attention, not asymptotically less attention.\n\n## MLA: Compressed Is Excellent, but Compressed Is Not Linear\n\nMQA, GQA, and MLA attack the number of bytes stored per token:\n\n**MQA** shares one KV head across query heads.**GQA** shares KV heads within groups.-\n**MLA** stores a compressed latent representation rather than full per-head keys and values.\n\nMLA can substantially reduce KV-cache capacity and bandwidth pressure. Hardware analyses show that the compact latent representation can move decode execution away from a bandwidth-bound regime, depending on how reconstruction and projection are implemented.\n\nThat is a major systems improvement.\n\nBut MLA still retains a representation for each token: `capacity = O(L)`\n\n. Dense MLA still incorporates all previous entries for each query: `decode compute = O(L) prefill compute = O(L²)`\n\n.\n\nMLA is compressed dense attention—not linear-time attention.\n\nA suitcase with vacuum-packed clothes still gets heavier as you add shirts.\n\n## Sparse Attention: Fewer Accesses, Same Attic\n\nSuppose each query attends to a fixed number *k* of earlier tokens. If those locations are already known, the main attention operation costs: `O(k) per decode step O(Lk) during prefill`\n\n. With fixed *k*, that is constant decode work and linear prefill work. But what happens to all the unselected tokens?\n\n### Sliding-Window Attention\n\nIf tokens older than a fixed window *w* can never be referenced again, their KVs can be evicted: `capacity = traffic = decode compute = O(w) = O(1)`\n\n. This is the clean version. The mechanism forgets old details and admits it.\n\n### Sparse Access to Arbitrary History\n\nIf a future query may select any previous position, every old representation may need to remain somewhere: `capacity = O(L)`\n\n. Yet each query may retrieve and process only *k* entries: `traffic = O(k) attention compute = O(k)`\n\n. BigBird-style patterns reduce the attention graph to a linear number of local, random, and global edges instead of evaluating all token pairs. But a linear-sized sparse graph does not automatically imply a constant-size autoregressive state.\n\nLarge memory capacity, small per-query traffic, and small attention compute.\n\nThe model keeps the whole attic but retrieves only one box.\n\n## DSA: Top-k Attention Does Not Make Top-k Discovery Free\n\nDeepSeek Sparse Attention is particularly interesting because it exposes a cost that sparse-attention descriptions often politely escort out of the room:\n\nHow do you find the k relevant tokens?\n\nDSA uses a lightweight “lightning indexer” to score preceding tokens. It then selects the top-k entries and applies the more expensive MLA attention operation only to those selected entries. The indexer has a small number of heads and can operate in FP8, making it much cheaper than full attention.\n\nThe expensive attention portion is fixed in *k*: `core attention compute = O(k)`\n\n. But the current indexer still scores the query against the preceding sequence: `indexer compute = O(L)`\n\n. The index representations and MLA cache entries remain token-indexed: `occupied capacity = O(L)`\n\n. A more accurate description is therefore:\n\nDSA performs O(L) cheap retrieval work to avoid O(L) expensive attention work.\n\nThat can be an outstanding trade. A linear FP8 index scan may be far cheaper than full MLA over a million tokens. Asymptotic notation does not pay the electricity bill; constants and arithmetic intensity matter.\n\nStill, a million cheap comparisons are not zero comparisons.\n\nDuring prefill, query-to-history index scoring can remain quadratic over the complete sequence, while selected core attention is linear for fixed *k*. NVIDIA’s DSA implementation similarly exposes index scoring, top-k selection, and sparse-attention kernels as separate components.\n\nTo obtain genuinely sublinear or constant decode retrieval, DSA would need a different index structure—perhaps hierarchical, hashed, clustered, or otherwise sublinear. That would introduce new update costs, approximation behavior, and hardware complexity.\n\nThere are no free lunches. There are only increasingly sophisticated meal plans.\n\n## Recurrent Linear Attention: The History Becomes a State\n\nKernel linear attention changes the algebra. The layer does not retain every key and value separately. It updates a sufficient statistic: `S`\n\n. For fixed feature dimensions:t = St-1 + f(kt)vtT\n\n```\ncapacity = O(1)\ndecode traffic = O(1)\ndecode compute = O(1)\nprefill compute = O(L)\n```\n\nPerformer approximates softmax attention through random features, while GLA, DeltaNet, Gated DeltaNet, and KDA use learned decay, gating, or delta-rule updates to manage a recurrent matrix state more effectively.\n\nMamba and related state-space models occupy a similar systems regime. They maintain fixed-size recurrent state and use hardware-aware parallel algorithms for training and prefill, producing computation that scales linearly with sequence length.\n\nThis sounds strictly superior—until we remember the suitcase.\n\nA dense KV cache retains explicit token-indexed information. A recurrent model compresses an arbitrarily long history into a finite state. Gating enables rapid forgetting, while delta updates enable targeted replacement, but the information capacity remains bounded by the state dimension.\n\nRecurrent attention eliminates cache growth by answering:\n\n“What should we remember?”Dense attention answers:“Yes.”\n\n## Kimi Linear: Three Linear Layers in a Trench Coat with MLA\n\nNow for the provocative part.\n\nA Kimi Delta Attention layer is genuinely linear-time in prefill and recurrent at decode. It maintains a fixed-size state rather than a token-indexed KV cache.\n\nBut **Kimi Linear, the complete architecture, is not asymptotically linear**.\n\nThe published model uses a 3:1 ratio of KDA to global MLA layers: three recurrent KDA layers followed by one dense MLA layer. The authors accurately call it a hybrid architecture and report up to a 75% reduction in KV-cache usage relative to an all-MLA baseline.\n\nThat is a very good engineering outcome.\n\nIt does not change the asymptotic complexity of the complete model.\n\nIf a fixed fraction *p > 0* of layers uses dense global attention, then:\n\n```\ncapacity        = O(pL)  + O(1) = O(L)\ndecode traffic  = O(pL)  + O(1) = O(L)\ndecode compute  = O(pL)  + O(1) = O(L)\nprefill compute = O(pL²) + O(L) = O(L²)\n```\n\nReducing the coefficient by four can produce enormous speedups. But: `(1/4)L² = O(L²)`\n\nBig-O notation is famously unmoved by branding.\n\nKDA is a linear recurrent attention mechanism. Kimi Linear is a hybrid architecture with mostly KDA layers and periodic dense MLA layers.As a complete model, it retains O(L) cache growth, O(L) decode attention cost, and O(L²) prefill attention cost.\n\n“Hybrid linear” is a reasonable architectural name. “Constant-time decoding” would not be a reasonable model-level complexity claim.\n\n## We Need Better Labels\n\nRather than asking whether an architecture is “linear,” papers and systems reports should use more precise terms:\n\n-\n**Prefill-linear:** total prefill arithmetic is O(L). -\n**Decode-constant:** arithmetic per generated token is O(1) in prior\n\ncontext length. -\n**State-constant:** persistent attention state is O(1) in L. -\n**Traffic-constant:** data movement per generated token is O(1) in L. -\n**Cache-compressed:** persistent state remains O(L), but bytes per\n\ntoken are reduced. -\n**Sparse-access:** each query uses only O(k) history entries. -\n**Linear-indexed:** selection scans O(L) compact index entries before\n\nsparse attention. -\n**Hybrid-linear:** recurrent layers coexist with dense layers;\n\nmodel-level complexity must be reported separately.\n\nThese terms are less glamorous than “linear attention.”\n\nThey also have the unfortunate property of telling us what the architecture actually does.\n\n## The Future Is a Matrix, Not a Leaderboard\n\nThere is no single road from quadratic attention to “linear attention.” There is a multidimensional design space.\n\n### Large Capacity, Small Accesses\n\nRetain an enormous external history but retrieve only a fixed number of entries for each query.\n\n### Fixed State, Constant Decode\n\nCompress the entire past into recurrent state, accepting bounded information capacity.\n\n### Cheap Linear Search, Sparse Expensive Attention\n\nScan a compact index over the history, then run full attention on only the best candidates—as in current DSA.\n\n### Local State Plus Occasional Global Access\n\nUse constant-cost local or recurrent mechanisms most of the time and periodically invoke global attention.\n\n### Hierarchical Memory\n\nKeep a small recurrent state on-chip, a larger compressed cache in HBM, and an enormous sparse history in host or storage memory.\n\nThese architectures may all be described as “efficient attention,” but they optimize different resources.\n\n## Conclusion: Show Us All Three Bills\n\nThe phrase **linear attention** has become too ambiguous to carry a serious systems argument by itself.\n\nAn architecture can:\n\n- have linear prefill computation but a substantial fixed state;\n- have constant selected-attention compute but linear retrieval work;\n- retain O(L) memory while loading only O(1) entries;\n- compress KV capacity without changing sequence-length exponents; or\n- use linear layers three-quarters of the time while remaining quadratic overall.\n\nSo the next time someone says, “Our model uses linear attention,” the correct response is not applause.\n\nLinear in what?\n\nAsk for:\n\n- occupied memory capacity;\n- memory traffic per token;\n- prefill computation;\n- decode computation;\n- selection or indexing cost; and\n- model-level complexity—not merely the complexity of its favorite layer.\n\nIf those quantities are separated, the confusion disappears.\n\nIf they are not, *linear* may be doing less mathematical analysis and more marketing.\n\nAnd unlike the KV cache, our patience does not need to grow with sequence length.\n\n## References\n\n-\nSun et al.,\n\nhttps://arxiv.org/abs/2507.19595\n\n“Efficient Attention Mechanisms for Large Language Models: A Survey”\n\n. -\nZhang et al.,\n\n[“Efficient Attention Methods: Hardware-efficient, Sparse, Compact, and Linear Attention”](https://attention-survey.github.io/files/Attention_Survey.pdf). -\nDao et al.,\n\n.\n\n“FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness”\n\n-\nGeens and Verhelst,\n\nhttps://arxiv.org/abs/2506.02523\n\n“Hardware-Centric Analysis of DeepSeek’s Multi-Head Latent Attention”\n\n. -\nZaheer et al.,\n\n.\n\n“Big Bird: Transformers for Longer Sequences”\n\n-\nDeepSeek-AI,\n\n.\n\n“DeepSeek-V3.2: Pushing the Frontier of Open Large Language Models”\n\n-\nNVIDIA,\n\n.\n\n“DeepSeek Sparse Attention—cuDNN Frontend”\n\n-\nChoromanski et al.,\n\n.\n\n“Rethinking Attention with Performers”\n\n-\nYang, Kautz, and Hatamizadeh,\n\nhttps://arxiv.org/abs/2412.06464\n\n“Gated Delta Networks: Improving Mamba2 with Delta Rule”\n\n. -\nGu and Dao,\n\n.\n\n“Mamba: Linear-Time Sequence Modeling with Selective State Spaces”\n\n-\nKimi Team,\n\n.\n\n“Kimi Linear: An Expressive, Efficient Attention Architecture”", "url": "https://wpnews.pro/news/let-s-stop-calling-everything-linear-attention", "canonical_source": "https://htor.inf.ethz.ch/blog/index.php/2026/08/09/lets-stop-calling-everything-linear-attention/", "published_at": "2026-08-10 07:33:43+00:00", "updated_at": "2026-08-10 07:46:15.253797+00:00", "lang": "en", "topics": ["machine-learning", "large-language-models", "ai-research"], "entities": ["FlashAttention"], "alternates": {"html": "https://wpnews.pro/news/let-s-stop-calling-everything-linear-attention", "markdown": "https://wpnews.pro/news/let-s-stop-calling-everything-linear-attention.md", "text": "https://wpnews.pro/news/let-s-stop-calling-everything-linear-attention.txt", "jsonld": "https://wpnews.pro/news/let-s-stop-calling-everything-linear-attention.jsonld"}}