{"slug": "linear-attention-visualized", "title": "Linear Attention, Visualized", "summary": "Moonshot released Kimi K3, a 2-trillion-parameter model with a 1-million-token context window, in July 2026, using its linear-attention variant Kimi Delta Attention (KDA). The model trails only Claude Fable 5 and GPT-5.6 Sol across its evaluation suite, according to Moonshot. KDA and NVIDIA's Nemotron 3, which combines softmax attention with Mamba-2 recurrent layers, both offer linear-time sequence mixing during training and fixed-state decoding, though with some loss of expressiveness.", "body_md": "In July 2026, Moonshot released [Kimi K3](https://www.kimi.com/blog/kimi-k3), a 2T-parameter model with a 1M-token context window. K3 uses Moonshot's linear-attention variant, Kimi Delta Attention (KDA). Moonshot reports that it trails only Claude Fable 5 and GPT-5.6 Sol across its evaluation suite.\n\nNVIDIA had already made a similar choice in [Nemotron 3](https://arxiv.org/abs/2512.20856), which combines softmax attention with Mamba-2 recurrent layers. KDA and Mamba-2 use different recurrences. Both offer linear-time sequence mixing during training and fixed-state decoding, although this efficiency comes with some loss of expressiveness.\n\nIn this post, we visualize the path from self-attention to the linear-attention variants used in recent models.\n\n## Self-attention recap\n\nGiven hidden states $H$, three learned projections produce queries, keys, and values:\n\nwhere $Q,K\\in\\mathbb R^{n\\times d_k}$ and $V,O\\in\\mathbb R^{n\\times d_v}$, with $n$ denoting the sequence length.\n\nEach query $q_t$ represents a linear functional on the key space:\n\nEvaluating it at $k_j$ produces the scalar score $\\langle q_t,k_j\\rangle$. Softmax normalizes these scores into weights, and $o_t$ is the weighted sum of the corresponding values.\n\n## Self-attention is quadratic\n\nThe attention operation uses two matrix multiplications over the sequence:\n\nMasking and softmax add $\\mathcal O(n^2)$ work, so the attention operation costs\n\nwhich is quadratic in the sequence length.\n\nA direct implementation stores the $n\\times n$ score matrix. FlashAttention avoids materializing this matrix in off-chip memory, but the computation remains quadratic.\n\n## Optional: Attention vs. FFN\n\nFor model width $d$ and FFN expansion factor $e$:\n\nFor $e=4$, the full attention layer (attention operation + input/output projections) costs more than the FFN when $n>2d$; the attention operation alone does so when $n>4d$.\n\n## Naïve linear attention\n\nThe quadratic cost comes from multiplying $Q$ by $K^{\\mathsf T}$ first, which produces an $n\\times n$ score matrix. If we remove softmax, matrix multiplication is associative, so we can change the order of computation:\n\nThe left-hand order compares every query with every key before mixing the values. The right-hand order first summarizes all key–value pairs in the $d\\times d$ matrix $K^{\\mathsf T}V$, then applies each query to that summary. It never constructs an $n\\times n$ matrix.\n\nThe two matrix multiplications now cost\n\nTogether, they require approximately $4nd^2=\\mathcal O(nd^2)$ work, which is linear rather than quadratic in the sequence length.\n\nThis computation is not causal: the shared summary $K^{\\mathsf T}V$ contains key–value pairs from the entire sequence, so every query can read the future. The causal version is\n\nAt first glance, the causal mask appears to prevent direct reassociation of the full matrix product. However, row $t$ of $O$ depends only on $q_t$, $K_{1:t}$, and $V_{1:t}$, as shown below. (Hover a row of $O$ to trace its prefix computation.)\n\nOnce the masked columns are removed, row $t$ becomes an ordinary matrix product over a prefix:\n\nWe call the product of the key and value prefixes the state $S_t$:\n\n$S_t$ is a fixed-size linear map from key space to value space. Moving from one token to the next adds exactly one outer product, so we can build this state recurrently instead of recomputing the entire prefix:\n\nEach recurrent step costs $\\mathcal O(d^2)$, so processing $n$ tokens costs $\\mathcal O(nd^2)$. The cost is linear in the sequence length, hence the name linear attention.\n\n## Optional: Linear attention as an RNN\n\nThis recurrence defines an RNN with a matrix-valued hidden state $S_t$. Each key–value pair updates $S_t$. The query $q_t$ reads $S_t$ to produce $o_t$. This recurrent view is the idea behind [ Transformers are RNNs: Fast Autoregressive Transformers with Linear Attention](https://arxiv.org/abs/2006.16236).\n\nThe recurrent form reduces the arithmetic complexity, but it uses GPUs inefficiently:\n\n**Large activation memory.** A direct autograd implementation saves every $d\\times d$ state $S_t$ for the backward pass. The $n$ saved states require $\\mathcal O(nd^2)$ memory.**Sequential token updates.** Computing $S_t$ requires $S_{t-1}$. This dependency prevents the GPU from updating different tokens in parallel.**Poor Tensor Core utilization.** Tensor Cores accelerate tiled matrix multiply-accumulate operations. The outer product $k_t^{\\mathsf T}v_t$ has shape $(d\\times1)(1\\times d)$, so its reduction dimension is only one. This shape does not fill the Tensor Core tiles along the reduction dimension.\n\n## Chunkwise algorithm\n\nTo solve these problems, we introduce the chunkwise algorithm. It uses the parallel form $\\bigl(QK^{\\mathsf T}\\odot M\\bigr)V$ within each chunk and the recurrent form $S_{\\mathrm{out}}=S_{\\mathrm{in}}+K^{\\mathsf T}V$ between chunks.\n\nAssume $C$ divides $n$. Split the sequence into $n/C$ chunks of $C$ consecutive tokens. Let $Q_r,K_r,V_r\\in\\mathbb R^{C\\times d}$ denote chunk $r$.\n\n### Block the causal matrix\n\nLet $A=QK^{\\mathsf T}\\odot M$. Partition $A$ into $C\\times C$ blocks and partition $V$ and $O$ into $C\\times d$ blocks. Let $A_{r,s}$ denote block $(r,s)$ of $A$. In block row $r$, blocks to the left contain earlier chunks, the diagonal block contains the current chunk, and blocks to the right contain future tokens.\n\n**A** n×n in C×C blocks\n\nThe output for chunk $r$ splits into history and local contributions:\n\nThe local block keeps a $C\\times C$ causal mask because tokens in the chunk have different prefix lengths. The history blocks need no mask because every token in chunk $r$ may read every token in an earlier chunk.\n\n### Compress the history\n\nLet $K_{<r}$ and $V_{<r}$ contain all chunks before chunk $r$. Associativity compresses the dense history:\n\n$S_r$ contains all key–value updates before chunk $r$. Let $M_c$ denote the $C\\times C$ causal mask inside the chunk. The chunk output is\n\n### Update the state\n\nAfter computing $O_r$, add the current chunk to the history:\n\nThe matrix product $K_r^{\\mathsf T}V_r$ batches $C$ rank-one updates into a dense GEMM. Its reduction dimension is $C$ rather than $1$, so it uses Tensor Cores efficiently.\n\nFor the recurrent state, autograd now saves one boundary state per chunk. These $n/C$ states require $\\mathcal O((n/C)d^2)$ memory. Chunk boundaries still form a sequential chain, but the chain now contains $n/C$ steps instead of $n$.\n\n### Cost\n\nEach chunk requires\n\nAcross $n/C$ chunks, the total cost is\n\nLarger chunks increase the local $nCd$ term but shorten the sequential state chain. $C=1$ recovers the recurrent form, while $C=n$ recovers the full parallel form. Every valid $C$ provides exact results. [FLA](https://github.com/fla-org/flash-linear-attention) uses $C=64$, while [FlashKDA](https://github.com/MoonshotAI/FlashKDA) uses $C=16$.\n\n## RetNet: scalar decay\n\nIn Naïve linear attention, each outer-product write $k_t^{\\mathsf T}v_t$ enters the state with coefficient one and never decays. New and old writes therefore receive the same age-independent coefficient. [RetNet](https://arxiv.org/abs/2307.08621) introduces a recency bias by shrinking the existing state before adding the current write:\n\n$\\gamma$ is a fixed scalar, so it scales the entire state uniformly. The figure follows one key–value write through the recurrence:\n\nA write from $s$ steps earlier has weight $\\gamma^s$. The weight decreases exponentially with age.\n\n### Parallel form\n\nTo compute the recurrence in parallel, note that a write at position $j$ undergoes $t-j$ decay steps before query $q_t$ reads it. Its coefficient is therefore $\\gamma^{t-j}$. Collect these coefficients in a decayed causal matrix:\n\nThe diagonal is one because the current write has not decayed. Moving one column to the left adds one decay step.\n\nThe parallel form replaces the binary causal mask $M$ with $\\Gamma$:\n\n### Chunkwise form\n\nThe chunkwise form again separates incoming history from interactions inside the chunk. Let $S_r$ denote the state immediately before chunk $r$, and let $\\ell=1,\\ldots,C$ index positions in the current chunk.\n\n#### Decay incoming history\n\nBefore $q_{r,\\ell}$ (the query at local position $\\ell$ in chunk $r$) reads $S_r$, the state crosses $\\ell$ decay steps. Define the boundary-to-query factors\n\nThe first illustration applies one entry of $\\lambda$ to each row of $Q_rS_r$. (Hover a row of $Q_r$ to trace its decay factor and the resulting history row.)\n\nWe can move the row scale to $Q_r$ and define the decayed query:\n\nThe left arrow denotes left-boundary decay: row $\\ell$ includes the factor $\\gamma^\\ell$ from the chunk boundary to $q_{r,\\ell}$.\n\nThis gives the history term\n\n#### Compute the local block\n\nPartitioning the full parallel computation into $C\\times C$ blocks exposes the same history and local split:\n\nT⊙ Γ)decayed scores · n×n → C×C blocks\n\n**local**\n\n**history**\n\nA write at local position $m$ crosses $\\ell-m$ decay steps before $q_{r,\\ell}$ reads it. The local decay matrix is\n\nThe local term is\n\n#### Update the state\n\nWe now have the chunk output $O_r$. Next, we construct the state passed to the next chunk. The incoming state crosses all $C$ steps, while a write at local position $\\ell$ crosses only $C-\\ell$ steps. Define the write-to-boundary factors\n\nEach token contributes one decayed outer-product write:\n\nFor $C=2$, the first equality expands the shared chunk dimension and applies $ρ_ℓ$ to deposit $ℓ$. The two partial updates then sum to $\\Delta S_r$. Hover a scale or partial update to trace the corresponding key column and value row.\n\nWe can then absorb these two scale factors into the corresponding rows of $K_r$:\n\nThe right arrow denotes right-boundary decay: key row $\\ell$ includes the factor $\\gamma^{C-\\ell}$ from its position to the chunk's right boundary.\n\n#### Complete chunkwise form\n\nCombining the history and local terms gives the complete chunkwise equations:\n\nThe arrows identify the relevant boundary. Queries absorb decay from the left boundary, while keys absorb decay toward the right boundary. The incoming history state decays from the left boundary to each query position; each key–value write made inside the chunk decays from its position to the right boundary. This exact regrouping keeps the same $\\mathcal O(nCd+nd^2)$ asymptotic cost as the Naïve chunkwise algorithm.\n\n## Mamba-2: input-dependent scalar decay\n\nRetNet applies the same decay $\\gamma$ at every position. [Mamba-2](https://arxiv.org/abs/2405.21060) replaces it with a token-dependent scalar $a_t$. Its recurrent form is\n\nEach token supplies one scalar for the entire state. A value near one preserves history; a value near zero nearly clears it.\n\n## Optional: Constructing the decay gate\n\nThe recurrence assumes $0<a_t<1$. During training, the model must learn $a_t$ while preserving this constraint. Mamba-2 constructs the gate by discretizing continuous-time exponential decay:\n\nOver one token interval $\\Delta_t$, the exact solution is\n\nFirst, Mamba-2 learns a positive decay rate. It stores an unconstrained log-rate $A_{\\log}$ and exponentiates it:\n\nThe rate $\\lambda$ specifies the decay **per unit time**. Discretization converts the continuous system into one transition per token. Mamba-2 computes an input-dependent step size $\\Delta_t$ for the transition at token $t$, so different tokens advance the continuous system by different amounts. A token with a larger $\\Delta_t$ advances the system farther and produces stronger decay; a token with a smaller $\\Delta_t$ produces weaker decay. From the token representation $h_t$, a learned projection $W_\\Delta$ produces an unconstrained step $\\widetilde\\Delta_t$. The model adds a learned bias $b_\\Delta$ and applies softplus to make the step positive:\n\nCombining the positive rate and interval gives\n\n### Parallel form\n\nThe outer-product write $k_j^{\\mathsf T}v_j$ enters the state at position $j$. To reach $S_t$, subsequent transitions multiply it by $a_{j+1},a_{j+2},\\ldots,a_t$. Its survival coefficient is\n\nCollect these survival coefficients in the decay matrix $\\Gamma$. Entry $\\Gamma[t,j]$ scales the score between $q_t$ and $k_j$. (Hover an entry of $\\Gamma$ to trace the corresponding query, key, value, and output.)\n\nFor an inclusive interval, define\n\nThe decay matrix becomes\n\nThe parallel equation keeps the same form as RetNet:\n\n### Chunkwise form\n\nThe chunkwise derivation follows RetNet. Each fixed power of $\\gamma$ is replaced by a product of token-dependent gates.\n\nWithin chunk $r$, let $a_{r,\\ell}$ denote the gate at local position $\\ell$, and define\n\n#### Decay incoming history\n\nBefore $q_{r,\\ell}$ reads the incoming state $S_r$, the recurrence multiplies $S_r$ by $\\lambda_r[1:\\ell]$. As in RetNet, absorb this scalar into the query row:\n\nThe left arrow again denotes decay from the chunk's left boundary to each query position.\n\n#### Compute the local block\n\nThe write $k_{r,m}^{\\mathsf T}v_{r,m}$ is multiplied by $\\lambda_r[m+1:\\ell]$ before it contributes to the output at local position $\\ell$. Therefore,\n\nThe local output is\n\n#### Update the state\n\nWe now have $O_r$. To construct the state for the next chunk, multiply the incoming state by $\\lambda_r[1:C]$. The write at local position $\\ell$ is multiplied only by the subsequent gates, whose product is $\\lambda_r[\\ell+1:C]$. Absorb this suffix product into its key: (Hover an element of $\\operatorname{diag}(\\lambda_r[\\ell+1:C])$ to trace the corresponding key column and value row.)\n\nThe right arrow denotes decay from each write position to the chunk's right boundary. The state update is\n\n#### Complete chunkwise form\n\nCombining the output and state equations gives\n\nMamba-2 therefore keeps RetNet's scalar state transition but lets each token choose how much history survives. The gate remains scalar, so it scales every entry of the state equally.\n\n## Optional: Discretization convention\n\nDiscretization affects both the decay and the new write. Mamba-2 also multiplies the write by the step size $\\Delta_t$, so the literal recurrence is\n\nTo keep the same notation as the preceding sections, define\n\nAll equations above use this rescaled $v_t$.\n\n## DeltaNet: correcting memory collisions\n\nMamba-2 decays the entire history at once. Sometimes we instead want to correct the association addressed by the current key without uniformly decaying the entire memory. [DeltaNet](https://arxiv.org/abs/2406.06484) provides this more selective update.\n\n### Why additive writes collide\n\nTreat $S$ as a key–value associative memory. For each association $(k_i,v_i)$, we would like to retrieve $v_i$ by reading the memory with $k_i$:\n\nHowever, for a unit-norm key $k_i$, the read contains\n\nThe first term is the intended value. The other terms come from keys that overlap with $k_i$. Orthogonal keys eliminate this interference, but learned keys generally are not mutually orthogonal. A raw outer-product write can therefore collide with associations already stored in $S$.\n\nDeltaNet first asks what value the current state associates with $k_t$:\n\nThe residual $v_t-\\widehat v_t$ points from this prediction toward the desired value $v_t$. DeltaNet writes a step in that direction at $k_t$:\n\n### An intuitive example\n\nConsider a simplified memory with two associations. The labels below stand for key and value vectors. Assume the Alice and Bob keys are orthogonal and unit norm, so a write at Bob's key does not change the value read at Alice's key.\n\n**Alice** at school\n\n**Bob** at home\n\n**Bob** at the office\n\nReading the state with Bob's key first returns “at home.” The target is “at the office,” so DeltaNet writes the difference between those two value vectors at Bob's key. Reading Bob after the update gives\n\nIf $\\beta=1$, the new value replaces the old prediction exactly. A smaller $\\beta$ makes a partial correction.\n\nMore generally, when $\\lVert k_t\\rVert_2=1$,\n\nThe unit-norm condition matters. Without it, reading the update back with $k_t$ introduces an additional factor $\\lVert k_t\\rVert_2^2$.\n\n## Optional: Test-time training (TTT) interpretation\n\nAnother interpretation treats $S$ as a small linear model that maps keys to values. The pair $(k_t,v_t)$ supplies one online training example with squared-error loss\n\nIts gradient is\n\nOne gradient step from $S_{t-1}$ with learning rate $\\beta_t$ gives\n\nwhich is exactly the DeltaNet update. This view connects the delta rule to [ Learning to (Learn at Test Time)](https://arxiv.org/abs/2407.04620): the forward pass updates the recurrent state as a fast-weight model.\n\nThe same view also recovers the preceding recurrences. With a unit learning rate:\n\n| Method | Online loss $\\mathcal L_t(S)$ | State update |\n|---|---|---|\n| Naïve | $-\\langle k_tS,v_t\\rangle$ | $S_t=S_{t-1}+k_t^{\\mathsf T}v_t$ |\n| RetNet | $-\\langle k_tS,v_t\\rangle+\\dfrac{1-\\gamma}{2}\\lVert S\\rVert_F^2$ | $S_t=\\gamma S_{t-1}+k_t^{\\mathsf T}v_t$ |\n| Mamba-2 | $-\\langle k_tS,v_t\\rangle+\\dfrac{1-a_t}{2}\\lVert S\\rVert_F^2$ | $S_t=a_tS_{t-1}+k_t^{\\mathsf T}v_t$ |\n\nFrom the TTT view, Naïve linear attention has an unbounded objective: increasing $S$ along $k_t^{\\mathsf T}v_t$ can drive the linear loss toward $-\\infty$. RetNet adds a fixed $L_2$ penalty on the state, while Mamba-2 makes the strength of that penalty token-dependent through $a_t$.\n\nThe example explains one token update. We now derive the recurrent and full-sequence matrix forms.\n\n### Recurrent form\n\nDefine the scaled correction\n\nThe complete token recurrence is\n\nOnce $u_t$ is known, the state has the same additive form as Naïve linear attention:\n\nUnlike the raw values $v_t$, the corrections $u_t$ are causally coupled: each $u_t$ depends on earlier corrections through $S_{t-1}$. We therefore cannot form all rows of $U$ independently as in Naïve linear attention. We first collect this dependency into a triangular system and solve for $U$; only then can we reuse the linear-attention readout.\n\n### The triangular correction system\n\nSubstitute\n\ninto the prediction. Then\n\nFor the first three tokens,\n\nOnly earlier corrections appear on the right. Define the strictly lower-triangular matrix\n\nLet $M$ be the inclusive causal mask and $M^-:=M-I$ its strictly lower-triangular part. Then\n\nStack $u_t$ into $U\\in\\mathbb R^{n\\times d}$ and define $B:=\\operatorname{diag}(\\beta)$. The row recurrences become (as you may verify):\n\nEquivalently,\n\n## Optional: Solving for U with forward substitution\n\nWe do not compute $(I+BL)^{-1}$ explicitly. Forming a dense $n\\times n$ inverse costs $\\mathcal O(n^3)$.\n\nSet $A:=I+BL$ and $R:=BV$. Since $A$ is unit lower triangular, **forward substitution** solves\n\none row at a time. At row $t$, the earlier rows of $U$ are already known, and $u_t$ is the only new row. (Hover a row to follow the known inputs and current unknown.)\n\nFor $U\\in\\mathbb R^{n\\times d}$, row $t$ costs $\\mathcal O(td)$ and the full solve costs $\\mathcal O(n^2d)$.\n\n### Parallel output\n\nAfter solving for $U$, the state is a sum of key–correction outer products. The causal output therefore reuses the Naïve linear-attention equation with $U$ in place of $V$:\n\n### Chunkwise algorithm\n\nThe full $n$-token triangular system still requires quadratic work. Divide the sequence into chunks of $C$ tokens instead. Each chunk starts from its incoming boundary state. Local dependencies then fit in a $C\\times C$ system, while a $d\\times d$ state carries history between chunks. This regrouping is exact.\n\n#### One chunk as an affine recurrence\n\nLet $r$ index chunks and $\\ell\\in{1,\\ldots,C}$ index tokens in the current chunk. $S_r$ is the state before the chunk, and $S_{r+1}$ is the state after it. Rewrite the DeltaNet update as\n\nFor example, a chunk with $C=2$ contains two local transitions:\n\nUse $\\Lambda_r[i:j]$ for the ordered transition product over local positions $i$ through $j$ in chunk $r$. Applying the lemma to the incoming state $S_r$ gives\n\nThe first term carries the incoming state across the chunk. $H_r$ is the within-chunk **convolution sum** of local writes. Computing either expression directly would require many dense $d\\times d$ transition products.\n\n#### Compact the transition product with WY\n\nFor DeltaNet, the two vectors in each DPLR correction are $k_{r,\\ell}$ and $\\beta_{r,\\ell}k_{r,\\ell}$. We absorb $\\beta_{r,\\ell}$ into the triangular factor below. WY then gives\n\nTo construct $W_r$, maintain the partial form\n\nThe first three rows of $W_r$ follow by substituting one transition at a time:\n\nThe general row is\n\nLet $M_c$ be the inclusive $C\\times C$ causal mask, $M_c^-:=M_c-I$, and\n\nAs in the full-sequence parallel form, stacking these row recurrences produces:\n\n#### Compact the local writes\n\nThe local convolution sum $H_r$ is the state produced by the tokens in chunk $r$ when the incoming state is zero. The full-sequence parallel form already showed that DeltaNet writes its state as $K^{\\mathsf T}U$. Applying the same result within the chunk gives\n\nThe first three rows of $U_r$ are (as you may verify)\n\nThe general row is\n\nThis is exactly the full-sequence correction recurrence, applied only within chunk $r$. Stacking the rows gives:\n\n## Optional: Use UT to share the triangular transform\n\nSolving for $W_r$ and $U_r$ separately would repeat the same lower-triangular solve. UT lets us solve once for a shared transform, then obtain both matrices with dense matrix multiplications.\n\n**UT representation.** The WY form leaves $W$ implicit. For the DPLR product above, let $V$ stack the right factors $v_1,\\ldots,v_m$. Because row $i$ of $W$ depends only on $v_1,\\ldots,v_i$, we can write\n\nfor a lower-triangular $T\\in\\mathbb R^{m\\times m}$. The WY representation then becomes\n\nThis equivalent form is the **UT representation**. It exposes the triangular transform from the original right factors $V$ to their accumulated form $W$.\n\nThe derivations of $W_r$ and $U_r$ produced\n\nThe coefficient matrix is identical; only the right-hand side changes. Following UT, define $T_r\\in\\mathbb R^{C\\times C}$ by\n\nRight multiplication by $K_r$ or $V_r$ then gives\n\nThe compact transition product and convolution sum are therefore\n\n#### Correct the incoming state\n\nSubstitute the compact forms of $\\Lambda_r[1:C]$ and $H_r$ into the unrolled recurrence:\n\nDefine the actual correction rows as\n\n$U_r$ contains the corrections obtained with an empty incoming state. $W_rS_r$ is the transformed contribution of the predictions from $S_r$. Their difference gives the corrections produced by the actual chunk:\n\nThe update now has the same additive form as Naïve linear attention. We can therefore reuse its chunkwise output formula with $Z_r$ in place of $V_r$:\n\n## Complete per-chunk algorithm\n\nFor each chunk $r$,\n\nThe local quantities $L_r,T_r,W_r,$ and $U_r$ can be computed for all chunks in parallel. The boundary pass then threads $S_r$ through the chunks to form $Z_r$, $O_r$, and $S_{r+1}$.\n\n## Gated DeltaNet (GDN)\n\n[Gated DeltaNet (GDN)](https://arxiv.org/abs/2412.06464) extends DeltaNet by adding a token-dependent scalar decay to the state transition:\n\nGDN adds scalar decay to DeltaNet in the same way that Mamba-2 adds scalar decay to Naïve linear attention. DeltaNet also has the same chunkwise form as Naïve linear attention after replacing $V_r$ with $Z_r$.\n\n| Method | Chunkwise readout | Chunkwise state update |\n|---|---|---|\n| Naïve | $O_r=Q_rS_r+(Q_rK_r^{\\mathsf T}\\odot M_c)V_r$ | $S_{r+1}=S_r+K_r^{\\mathsf T}V_r$ |\n| Mamba-2 | $O_r=\\overleftarrow Q_rS_r+(Q_rK_r^{\\mathsf T}\\odot\\Gamma_{C,r})V_r$ | $S_{r+1}=\\lambda_r[1:C]S_r+\\overrightarrow K_r^{\\mathsf T}V_r$ |\n| DeltaNet | $O_r=Q_rS_r+(Q_rK_r^{\\mathsf T}\\odot M_c)Z_r$ | $S_{r+1}=S_r+K_r^{\\mathsf T}Z_r$ |\n| GDN | $O_r=\\overleftarrow Q_rS_r+(Q_rK_r^{\\mathsf T}\\odot\\Gamma_{C,r})Z_r$ | $S_{r+1}=\\lambda_r[1:C]S_r+\\overrightarrow K_r^{\\mathsf T}Z_r$ |\n\n## Complete per-chunk algorithm\n\nFor each chunk $r$, define the decay products and the correction gate:\n\n## Gated Delta Product (GDP)\n\n[Gated Delta Product (GDP)](https://arxiv.org/abs/2502.10297) replaces GDN's single generalized Householder transition per token with an ordered product of $n_h$ transitions.\n\nEach transition has its own key, value, and correction gate, all projected from the same input token. The transitions act along different learned key directions, increasing computation and expressiveness without changing the shape of $S_t$.\n\nFor substep $j$, define\n\nFor example, with $n_h=3$, the token update is\n\nThe decay $a_t$ is applied before the first substep, so the previous state decays only once. The query reads the state only after all three updates have finished.\n\n### GDP as GDN over a virtual sequence\n\nGDP can be written as an ordinary GDN recurrence over a longer sequence. Treat its $n_h$ substeps as consecutive positions in that sequence. We call the original positions **real tokens** and the expanded positions **virtual steps**.\n\nFor $n_h=3$, one real token expands into three virtual steps. Two real tokens therefore become six:\n\n| real token $t=1$ | real token $t=2$ | |||||\n|---|---|---|---|---|---|---|\n| virtual step $m$ | 1 | 2 | 3 | 4 | 5 | 6 |\n| substep $(t,j)$ | $(1,1)$ | $(1,2)$ | $(1,3)$ | $(2,1)$ | $(2,2)$ | $(2,3)$ |\n| $\\tilde k_m$ | $k_{1,1}$ | $k_{1,2}$ | $k_{1,3}$ | $k_{2,1}$ | $k_{2,2}$ | $k_{2,3}$ |\n| $\\tilde v_m$ | $v_{1,1}$ | $v_{1,2}$ | $v_{1,3}$ | $v_{2,1}$ | $v_{2,2}$ | $v_{2,3}$ |\n| $\\tilde\\beta_m$ | $\\beta_{1,1}$ | $\\beta_{1,2}$ | $\\beta_{1,3}$ | $\\beta_{2,1}$ | $\\beta_{2,2}$ | $\\beta_{2,3}$ |\n| $\\tilde a_m$ | $a_1$ | 1 | 1 | $a_2$ | 1 | 1 |\n| $\\tilde q_m$ | 0 | 0 | $q_1$ | 0 | 0 | $q_2$ |\n| retained output | — | — | $o_1$ | — | — | $o_2$ |\n\nThe decay appears at the first virtual step of each real token; the remaining steps use $1$. The query appears at the last virtual step; the earlier steps use zero queries because they do not produce outputs.\n\nTo write this construction generally, flatten token $t$ and substep $j$ into\n\nMap the GDP inputs to this virtual sequence by setting\n\nEvery virtual step then follows the GDN recurrence:\n\nFlattening preserves the update order. We can therefore run the GDN chunkwise algorithm on a virtual sequence of length $nn_h$ and retain every $n_h$-th output. Only the sequence becomes longer; the recurrent state keeps the same shape.\n\n## Gated Linear Attention (GLA)\n\nMamba-2 lets each token choose how much of the previous state survives, but its gate is scalar:\n\nThe same $a_t$ scales every row of $S_{t-1}$. It can shorten or extend the lifetime of the memory as a whole, but it cannot preserve one channel while rapidly forgetting another.\n\n[Gated Linear Attention (GLA)](https://arxiv.org/abs/2312.06635) replaces the scalar gate with a vector\n\nand uses the recurrence\n\nThe two gates scale the same state differently:\n\nGLA therefore gives each channel its own forgetting rate.\n\n### Chunkwise form\n\nMultiplying diagonal decay matrices is equivalent to multiplying their diagonal vectors elementwise:\n\nWe therefore replace Mamba-2's scalar prefix product with a vector prefix product. Within chunk $r$, define\n\nHere, $\\bigodot$ denotes a repeated Hadamard (elementwise) product.\n\nAfter a write enters the state, later tokens decay each of its channels independently:\n\n**Incoming state.** The incoming state reaches local position $\\ell$ through $\\boldsymbol\\lambda_r[1:\\ell]$. Stack these vectors and absorb them into $Q_r$:\n\n**Local writes.** A write at position $m$ reaches position $\\ell$ through $\\boldsymbol\\lambda_r[m+1:\\ell]$. Its local score is therefore\n\nFor the state update, absorb the decay from each write to the right boundary into its key:\n\nThe complete chunkwise equations are\n\n## Kimi Delta Attention (KDA)\n\n[Kimi Delta Attention (KDA)](https://arxiv.org/abs/2510.26692) adds GLA's channelwise decay to GDN's delta rule:\n\nKDA first decays the state channel by channel, then applies the delta correction. If $\\boldsymbol a_t=a_t\\boldsymbol 1$, it reduces exactly to GDN.\n\n### Reuse GLA and GDN\n\nThe GLA and GDN sections give most of the chunkwise algorithm. GLA gives the decayed matrices $\\overleftarrow Q_r$ and $\\overrightarrow K_r$, together with the local score matrix $A_r$. GDN shows that the delta rule replaces the value rows $V_r$ with correction rows $Z_r$. Therefore,\n\nIt remains to compute $Z_r$. KDA reuses GDN's correction solve but replaces scalar decay with channelwise decay. In this solve, decay appears in two quantities: $L_r$ and $\\overleftarrow K_r$.\n\nKDA keeps this structure and replaces the scalar decay products with channelwise products:\n\n| Quantity | GDN: scalar decay | KDA: channelwise decay |\n|---|---|---|\n| Decay from $i$ through $j$ | $\\lambda_r[i:j]=\\prod_{s=i}^{j}a_{r,s}$ | $\\boldsymbol\\lambda_r[i:j]=\\bigodot_{s=i}^{j}\\boldsymbol a_{r,s}$ |\n| Incoming prediction key | $\\overleftarrow k_{r,\\ell}=\\lambda_r[1:\\ell]k_{r,\\ell}$ | $\\overleftarrow k_{r,\\ell}=\\boldsymbol\\lambda_r[1:\\ell]\\odot k_{r,\\ell}$ |\n| $L_r[\\ell,m]$ for $m<\\ell$ | $\\lambda_r[m+1:\\ell](k_{r,\\ell}k_{r,m}^{\\mathsf T})$ | $k_{r,\\ell}\\bigl(\\boldsymbol\\lambda_r[m+1:\\ell]\\odot k_{r,m}\\bigr)^{\\mathsf T}$ |\n\nUsing the new $L_r$ and $\\overleftarrow K_r$, reuse GDN's triangular transform. Let $B_r:=\\operatorname{diag}(\\beta_{r,1},\\ldots,\\beta_{r,C})$. Then\n\n## Complete per-chunk algorithm\n\nFor each chunk $r$, define the channelwise decay products and correction gate:\n\nStack the transformed rows into $\\overleftarrow Q_r$, $\\overleftarrow K_r$, and $\\overrightarrow K_r$. Define the two causal within-chunk matrices:\n\n## Sources and further reading\n\n### Related papers\n\n[ Transformers are RNNs](https://arxiv.org/abs/2006.16236),\n\n[,](https://arxiv.org/abs/2307.08621)\n\n*RetNet*[,](https://arxiv.org/abs/2312.06635)\n\n*Gated Linear Attention (GLA)*[,](https://arxiv.org/abs/2405.21060)\n\n*Mamba-2 / SSD*[,](https://arxiv.org/abs/2406.06484)\n\n*DeltaNet*[,](https://arxiv.org/abs/2412.06464)\n\n*Gated DeltaNet (GDN)*[, and](https://arxiv.org/abs/2502.10297)\n\n*DeltaProduct (GDP)*\n\n*Kimi Linear / Kimi Delta Attention (KDA)*", "url": "https://wpnews.pro/news/linear-attention-visualized", "canonical_source": "https://snowchord.com/blog/linear-attention-visualized/", "published_at": "2026-07-28 17:30:12+00:00", "updated_at": "2026-07-28 17:52:37.627643+00:00", "lang": "en", "topics": ["artificial-intelligence", "large-language-models", "ai-research", "ai-products"], "entities": ["Moonshot", "Kimi K3", "Kimi Delta Attention", "NVIDIA", "Nemotron 3", "Mamba-2", "Claude Fable 5", "GPT-5.6 Sol"], "alternates": {"html": "https://wpnews.pro/news/linear-attention-visualized", "markdown": "https://wpnews.pro/news/linear-attention-visualized.md", "text": "https://wpnews.pro/news/linear-attention-visualized.txt", "jsonld": "https://wpnews.pro/news/linear-attention-visualized.jsonld"}}