Linear Attention, Visualized

Moonshot released Kimi K3, a 2-trillion-parameter model with a 1-million-token context window, in July 2026, using its linear-attention variant Kimi Delta Attention (KDA). The model trails only Claude Fable 5 and GPT-5.6 Sol across its evaluation suite, according to Moonshot. KDA and NVIDIA's Nemotron 3, which combines softmax attention with Mamba-2 recurrent layers, both offer linear-time sequence mixing during training and fixed-state decoding, though with some loss of expressiveness.

In July 2026, Moonshot released Kimi K3 https://www.kimi.com/blog/kimi-k3 , a 2T-parameter model with a 1M-token context window. K3 uses Moonshot's linear-attention variant, Kimi Delta Attention KDA . Moonshot reports that it trails only Claude Fable 5 and GPT-5.6 Sol across its evaluation suite. NVIDIA had already made a similar choice in Nemotron 3 https://arxiv.org/abs/2512.20856 , which combines softmax attention with Mamba-2 recurrent layers. KDA and Mamba-2 use different recurrences. Both offer linear-time sequence mixing during training and fixed-state decoding, although this efficiency comes with some loss of expressiveness. In this post, we visualize the path from self-attention to the linear-attention variants used in recent models. Self-attention recap Given hidden states $H$, three learned projections produce queries, keys, and values: where $Q,K\in\mathbb R^{n\times d k}$ and $V,O\in\mathbb R^{n\times d v}$, with $n$ denoting the sequence length. Each query $q t$ represents a linear functional on the key space: Evaluating it at $k j$ produces the scalar score $\langle q t,k j\rangle$. Softmax normalizes these scores into weights, and $o t$ is the weighted sum of the corresponding values. Self-attention is quadratic The attention operation uses two matrix multiplications over the sequence: Masking and softmax add $\mathcal O n^2 $ work, so the attention operation costs which is quadratic in the sequence length. A direct implementation stores the $n\times n$ score matrix. FlashAttention avoids materializing this matrix in off-chip memory, but the computation remains quadratic. Optional: Attention vs. FFN For model width $d$ and FFN expansion factor $e$: For $e=4$, the full attention layer attention operation + input/output projections costs more than the FFN when $n 2d$; the attention operation alone does so when $n 4d$. Naïve linear attention The quadratic cost comes from multiplying $Q$ by $K^{\mathsf T}$ first, which produces an $n\times n$ score matrix. If we remove softmax, matrix multiplication is associative, so we can change the order of computation: The left-hand order compares every query with every key before mixing the values. The right-hand order first summarizes all key–value pairs in the $d\times d$ matrix $K^{\mathsf T}V$, then applies each query to that summary. It never constructs an $n\times n$ matrix. The two matrix multiplications now cost Together, they require approximately $4nd^2=\mathcal O nd^2 $ work, which is linear rather than quadratic in the sequence length. This computation is not causal: the shared summary $K^{\mathsf T}V$ contains key–value pairs from the entire sequence, so every query can read the future. The causal version is At first glance, the causal mask appears to prevent direct reassociation of the full matrix product. However, row $t$ of $O$ depends only on $q t$, $K {1:t}$, and $V {1:t}$, as shown below. Hover a row of $O$ to trace its prefix computation. Once the masked columns are removed, row $t$ becomes an ordinary matrix product over a prefix: We call the product of the key and value prefixes the state $S t$: $S t$ is a fixed-size linear map from key space to value space. Moving from one token to the next adds exactly one outer product, so we can build this state recurrently instead of recomputing the entire prefix: Each recurrent step costs $\mathcal O d^2 $, so processing $n$ tokens costs $\mathcal O nd^2 $. The cost is linear in the sequence length, hence the name linear attention. Optional: Linear attention as an RNN This recurrence defines an RNN with a matrix-valued hidden state $S t$. Each key–value pair updates $S t$. The query $q t$ reads $S t$ to produce $o t$. This recurrent view is the idea behind Transformers are RNNs: Fast Autoregressive Transformers with Linear Attention https://arxiv.org/abs/2006.16236 . The recurrent form reduces the arithmetic complexity, but it uses GPUs inefficiently: Large activation memory. A direct autograd implementation saves every $d\times d$ state $S t$ for the backward pass. The $n$ saved states require $\mathcal O nd^2 $ memory. Sequential token updates. Computing $S t$ requires $S {t-1}$. This dependency prevents the GPU from updating different tokens in parallel. Poor Tensor Core utilization. Tensor Cores accelerate tiled matrix multiply-accumulate operations. The outer product $k t^{\mathsf T}v t$ has shape $ d\times1 1\times d $, so its reduction dimension is only one. This shape does not fill the Tensor Core tiles along the reduction dimension. Chunkwise algorithm To solve these problems, we introduce the chunkwise algorithm. It uses the parallel form $\bigl QK^{\mathsf T}\odot M\bigr V$ within each chunk and the recurrent form $S {\mathrm{out}}=S {\mathrm{in}}+K^{\mathsf T}V$ between chunks. Assume $C$ divides $n$. Split the sequence into $n/C$ chunks of $C$ consecutive tokens. Let $Q r,K r,V r\in\mathbb R^{C\times d}$ denote chunk $r$. Block the causal matrix Let $A=QK^{\mathsf T}\odot M$. Partition $A$ into $C\times C$ blocks and partition $V$ and $O$ into $C\times d$ blocks. Let $A {r,s}$ denote block $ r,s $ of $A$. In block row $r$, blocks to the left contain earlier chunks, the diagonal block contains the current chunk, and blocks to the right contain future tokens. A n×n in C×C blocks The output for chunk $r$ splits into history and local contributions: The local block keeps a $C\times C$ causal mask because tokens in the chunk have different prefix lengths. The history blocks need no mask because every token in chunk $r$ may read every token in an earlier chunk. Compress the history Let $K {<r}$ and $V {<r}$ contain all chunks before chunk $r$. Associativity compresses the dense history: $S r$ contains all key–value updates before chunk $r$. Let $M c$ denote the $C\times C$ causal mask inside the chunk. The chunk output is Update the state After computing $O r$, add the current chunk to the history: The matrix product $K r^{\mathsf T}V r$ batches $C$ rank-one updates into a dense GEMM. Its reduction dimension is $C$ rather than $1$, so it uses Tensor Cores efficiently. For the recurrent state, autograd now saves one boundary state per chunk. These $n/C$ states require $\mathcal O n/C d^2 $ memory. Chunk boundaries still form a sequential chain, but the chain now contains $n/C$ steps instead of $n$. Cost Each chunk requires Across $n/C$ chunks, the total cost is Larger chunks increase the local $nCd$ term but shorten the sequential state chain. $C=1$ recovers the recurrent form, while $C=n$ recovers the full parallel form. Every valid $C$ provides exact results. FLA https://github.com/fla-org/flash-linear-attention uses $C=64$, while FlashKDA https://github.com/MoonshotAI/FlashKDA uses $C=16$. RetNet: scalar decay In Naïve linear attention, each outer-product write $k t^{\mathsf T}v t$ enters the state with coefficient one and never decays. New and old writes therefore receive the same age-independent coefficient. RetNet https://arxiv.org/abs/2307.08621 introduces a recency bias by shrinking the existing state before adding the current write: $\gamma$ is a fixed scalar, so it scales the entire state uniformly. The figure follows one key–value write through the recurrence: A write from $s$ steps earlier has weight $\gamma^s$. The weight decreases exponentially with age. Parallel form To compute the recurrence in parallel, note that a write at position $j$ undergoes $t-j$ decay steps before query $q t$ reads it. Its coefficient is therefore $\gamma^{t-j}$. Collect these coefficients in a decayed causal matrix: The diagonal is one because the current write has not decayed. Moving one column to the left adds one decay step. The parallel form replaces the binary causal mask $M$ with $\Gamma$: Chunkwise form The chunkwise form again separates incoming history from interactions inside the chunk. Let $S r$ denote the state immediately before chunk $r$, and let $\ell=1,\ldots,C$ index positions in the current chunk. Decay incoming history Before $q {r,\ell}$ the query at local position $\ell$ in chunk $r$ reads $S r$, the state crosses $\ell$ decay steps. Define the boundary-to-query factors The first illustration applies one entry of $\lambda$ to each row of $Q rS r$. Hover a row of $Q r$ to trace its decay factor and the resulting history row. We can move the row scale to $Q r$ and define the decayed query: The left arrow denotes left-boundary decay: row $\ell$ includes the factor $\gamma^\ell$ from the chunk boundary to $q {r,\ell}$. This gives the history term Compute the local block Partitioning the full parallel computation into $C\times C$ blocks exposes the same history and local split: T⊙ Γ decayed scores · n×n → C×C blocks local history A write at local position $m$ crosses $\ell-m$ decay steps before $q {r,\ell}$ reads it. The local decay matrix is The local term is Update the state We now have the chunk output $O r$. Next, we construct the state passed to the next chunk. The incoming state crosses all $C$ steps, while a write at local position $\ell$ crosses only $C-\ell$ steps. Define the write-to-boundary factors Each token contributes one decayed outer-product write: For $C=2$, the first equality expands the shared chunk dimension and applies $ρ ℓ$ to deposit $ℓ$. The two partial updates then sum to $\Delta S r$. Hover a scale or partial update to trace the corresponding key column and value row. We can then absorb these two scale factors into the corresponding rows of $K r$: The right arrow denotes right-boundary decay: key row $\ell$ includes the factor $\gamma^{C-\ell}$ from its position to the chunk's right boundary. Complete chunkwise form Combining the history and local terms gives the complete chunkwise equations: The arrows identify the relevant boundary. Queries absorb decay from the left boundary, while keys absorb decay toward the right boundary. The incoming history state decays from the left boundary to each query position; each key–value write made inside the chunk decays from its position to the right boundary. This exact regrouping keeps the same $\mathcal O nCd+nd^2 $ asymptotic cost as the Naïve chunkwise algorithm. Mamba-2: input-dependent scalar decay RetNet applies the same decay $\gamma$ at every position. Mamba-2 https://arxiv.org/abs/2405.21060 replaces it with a token-dependent scalar $a t$. Its recurrent form is Each token supplies one scalar for the entire state. A value near one preserves history; a value near zero nearly clears it. Optional: Constructing the decay gate The recurrence assumes $0<a t<1$. During training, the model must learn $a t$ while preserving this constraint. Mamba-2 constructs the gate by discretizing continuous-time exponential decay: Over one token interval $\Delta t$, the exact solution is First, Mamba-2 learns a positive decay rate. It stores an unconstrained log-rate $A {\log}$ and exponentiates it: The rate $\lambda$ specifies the decay per unit time . Discretization converts the continuous system into one transition per token. Mamba-2 computes an input-dependent step size $\Delta t$ for the transition at token $t$, so different tokens advance the continuous system by different amounts. A token with a larger $\Delta t$ advances the system farther and produces stronger decay; a token with a smaller $\Delta t$ produces weaker decay. From the token representation $h t$, a learned projection $W \Delta$ produces an unconstrained step $\widetilde\Delta t$. The model adds a learned bias $b \Delta$ and applies softplus to make the step positive: Combining the positive rate and interval gives Parallel form The outer-product write $k j^{\mathsf T}v j$ enters the state at position $j$. To reach $S t$, subsequent transitions multiply it by $a {j+1},a {j+2},\ldots,a t$. Its survival coefficient is Collect these survival coefficients in the decay matrix $\Gamma$. Entry $\Gamma t,j $ scales the score between $q t$ and $k j$. Hover an entry of $\Gamma$ to trace the corresponding query, key, value, and output. For an inclusive interval, define The decay matrix becomes The parallel equation keeps the same form as RetNet: Chunkwise form The chunkwise derivation follows RetNet. Each fixed power of $\gamma$ is replaced by a product of token-dependent gates. Within chunk $r$, let $a {r,\ell}$ denote the gate at local position $\ell$, and define Decay incoming history Before $q {r,\ell}$ reads the incoming state $S r$, the recurrence multiplies $S r$ by $\lambda r 1:\ell $. As in RetNet, absorb this scalar into the query row: The left arrow again denotes decay from the chunk's left boundary to each query position. Compute the local block The write $k {r,m}^{\mathsf T}v {r,m}$ is multiplied by $\lambda r m+1:\ell $ before it contributes to the output at local position $\ell$. Therefore, The local output is Update the state We now have $O r$. To construct the state for the next chunk, multiply the incoming state by $\lambda r 1:C $. The write at local position $\ell$ is multiplied only by the subsequent gates, whose product is $\lambda r \ell+1:C $. Absorb this suffix product into its key: Hover an element of $\operatorname{diag} \lambda r \ell+1:C $ to trace the corresponding key column and value row. The right arrow denotes decay from each write position to the chunk's right boundary. The state update is Complete chunkwise form Combining the output and state equations gives Mamba-2 therefore keeps RetNet's scalar state transition but lets each token choose how much history survives. The gate remains scalar, so it scales every entry of the state equally. Optional: Discretization convention Discretization affects both the decay and the new write. Mamba-2 also multiplies the write by the step size $\Delta t$, so the literal recurrence is To keep the same notation as the preceding sections, define All equations above use this rescaled $v t$. DeltaNet: correcting memory collisions Mamba-2 decays the entire history at once. Sometimes we instead want to correct the association addressed by the current key without uniformly decaying the entire memory. DeltaNet https://arxiv.org/abs/2406.06484 provides this more selective update. Why additive writes collide Treat $S$ as a key–value associative memory. For each association $ k i,v i $, we would like to retrieve $v i$ by reading the memory with $k i$: However, for a unit-norm key $k i$, the read contains The first term is the intended value. The other terms come from keys that overlap with $k i$. Orthogonal keys eliminate this interference, but learned keys generally are not mutually orthogonal. A raw outer-product write can therefore collide with associations already stored in $S$. DeltaNet first asks what value the current state associates with $k t$: The residual $v t-\widehat v t$ points from this prediction toward the desired value $v t$. DeltaNet writes a step in that direction at $k t$: An intuitive example Consider a simplified memory with two associations. The labels below stand for key and value vectors. Assume the Alice and Bob keys are orthogonal and unit norm, so a write at Bob's key does not change the value read at Alice's key. Alice at school Bob at home Bob at the office Reading the state with Bob's key first returns “at home.” The target is “at the office,” so DeltaNet writes the difference between those two value vectors at Bob's key. Reading Bob after the update gives If $\beta=1$, the new value replaces the old prediction exactly. A smaller $\beta$ makes a partial correction. More generally, when $\lVert k t\rVert 2=1$, The unit-norm condition matters. Without it, reading the update back with $k t$ introduces an additional factor $\lVert k t\rVert 2^2$. Optional: Test-time training TTT interpretation Another interpretation treats $S$ as a small linear model that maps keys to values. The pair $ k t,v t $ supplies one online training example with squared-error loss Its gradient is One gradient step from $S {t-1}$ with learning rate $\beta t$ gives which is exactly the DeltaNet update. This view connects the delta rule to Learning to Learn at Test Time https://arxiv.org/abs/2407.04620 : the forward pass updates the recurrent state as a fast-weight model. The same view also recovers the preceding recurrences. With a unit learning rate: | Method | Online loss $\mathcal L t S $ | State update | |---|---|---| | Naïve | $-\langle k tS,v t\rangle$ | $S t=S {t-1}+k t^{\mathsf T}v t$ | | RetNet | $-\langle k tS,v t\rangle+\dfrac{1-\gamma}{2}\lVert S\rVert F^2$ | $S t=\gamma S {t-1}+k t^{\mathsf T}v t$ | | Mamba-2 | $-\langle k tS,v t\rangle+\dfrac{1-a t}{2}\lVert S\rVert F^2$ | $S t=a tS {t-1}+k t^{\mathsf T}v t$ | From the TTT view, Naïve linear attention has an unbounded objective: increasing $S$ along $k t^{\mathsf T}v t$ can drive the linear loss toward $-\infty$. RetNet adds a fixed $L 2$ penalty on the state, while Mamba-2 makes the strength of that penalty token-dependent through $a t$. The example explains one token update. We now derive the recurrent and full-sequence matrix forms. Recurrent form Define the scaled correction The complete token recurrence is Once $u t$ is known, the state has the same additive form as Naïve linear attention: Unlike the raw values $v t$, the corrections $u t$ are causally coupled: each $u t$ depends on earlier corrections through $S {t-1}$. We therefore cannot form all rows of $U$ independently as in Naïve linear attention. We first collect this dependency into a triangular system and solve for $U$; only then can we reuse the linear-attention readout. The triangular correction system Substitute into the prediction. Then For the first three tokens, Only earlier corrections appear on the right. Define the strictly lower-triangular matrix Let $M$ be the inclusive causal mask and $M^-:=M-I$ its strictly lower-triangular part. Then Stack $u t$ into $U\in\mathbb R^{n\times d}$ and define $B:=\operatorname{diag} \beta $. The row recurrences become as you may verify : Equivalently, Optional: Solving for U with forward substitution We do not compute $ I+BL ^{-1}$ explicitly. Forming a dense $n\times n$ inverse costs $\mathcal O n^3 $. Set $A:=I+BL$ and $R:=BV$. Since $A$ is unit lower triangular, forward substitution solves one row at a time. At row $t$, the earlier rows of $U$ are already known, and $u t$ is the only new row. Hover a row to follow the known inputs and current unknown. For $U\in\mathbb R^{n\times d}$, row $t$ costs $\mathcal O td $ and the full solve costs $\mathcal O n^2d $. Parallel output After solving for $U$, the state is a sum of key–correction outer products. The causal output therefore reuses the Naïve linear-attention equation with $U$ in place of $V$: Chunkwise algorithm The full $n$-token triangular system still requires quadratic work. Divide the sequence into chunks of $C$ tokens instead. Each chunk starts from its incoming boundary state. Local dependencies then fit in a $C\times C$ system, while a $d\times d$ state carries history between chunks. This regrouping is exact. One chunk as an affine recurrence Let $r$ index chunks and $\ell\in{1,\ldots,C}$ index tokens in the current chunk. $S r$ is the state before the chunk, and $S {r+1}$ is the state after it. Rewrite the DeltaNet update as For example, a chunk with $C=2$ contains two local transitions: Use $\Lambda r i:j $ for the ordered transition product over local positions $i$ through $j$ in chunk $r$. Applying the lemma to the incoming state $S r$ gives The first term carries the incoming state across the chunk. $H r$ is the within-chunk convolution sum of local writes. Computing either expression directly would require many dense $d\times d$ transition products. Compact the transition product with WY For DeltaNet, the two vectors in each DPLR correction are $k {r,\ell}$ and $\beta {r,\ell}k {r,\ell}$. We absorb $\beta {r,\ell}$ into the triangular factor below. WY then gives To construct $W r$, maintain the partial form The first three rows of $W r$ follow by substituting one transition at a time: The general row is Let $M c$ be the inclusive $C\times C$ causal mask, $M c^-:=M c-I$, and As in the full-sequence parallel form, stacking these row recurrences produces: Compact the local writes The local convolution sum $H r$ is the state produced by the tokens in chunk $r$ when the incoming state is zero. The full-sequence parallel form already showed that DeltaNet writes its state as $K^{\mathsf T}U$. Applying the same result within the chunk gives The first three rows of $U r$ are as you may verify The general row is This is exactly the full-sequence correction recurrence, applied only within chunk $r$. Stacking the rows gives: Optional: Use UT to share the triangular transform Solving for $W r$ and $U r$ separately would repeat the same lower-triangular solve. UT lets us solve once for a shared transform, then obtain both matrices with dense matrix multiplications. UT representation. The WY form leaves $W$ implicit. For the DPLR product above, let $V$ stack the right factors $v 1,\ldots,v m$. Because row $i$ of $W$ depends only on $v 1,\ldots,v i$, we can write for a lower-triangular $T\in\mathbb R^{m\times m}$. The WY representation then becomes This equivalent form is the UT representation . It exposes the triangular transform from the original right factors $V$ to their accumulated form $W$. The derivations of $W r$ and $U r$ produced The coefficient matrix is identical; only the right-hand side changes. Following UT, define $T r\in\mathbb R^{C\times C}$ by Right multiplication by $K r$ or $V r$ then gives The compact transition product and convolution sum are therefore Correct the incoming state Substitute the compact forms of $\Lambda r 1:C $ and $H r$ into the unrolled recurrence: Define the actual correction rows as $U r$ contains the corrections obtained with an empty incoming state. $W rS r$ is the transformed contribution of the predictions from $S r$. Their difference gives the corrections produced by the actual chunk: The update now has the same additive form as Naïve linear attention. We can therefore reuse its chunkwise output formula with $Z r$ in place of $V r$: Complete per-chunk algorithm For each chunk $r$, The local quantities $L r,T r,W r,$ and $U r$ can be computed for all chunks in parallel. The boundary pass then threads $S r$ through the chunks to form $Z r$, $O r$, and $S {r+1}$. Gated DeltaNet GDN Gated DeltaNet GDN https://arxiv.org/abs/2412.06464 extends DeltaNet by adding a token-dependent scalar decay to the state transition: GDN adds scalar decay to DeltaNet in the same way that Mamba-2 adds scalar decay to Naïve linear attention. DeltaNet also has the same chunkwise form as Naïve linear attention after replacing $V r$ with $Z r$. | Method | Chunkwise readout | Chunkwise state update | |---|---|---| | Naïve | $O r=Q rS r+ Q rK r^{\mathsf T}\odot M c V r$ | $S {r+1}=S r+K r^{\mathsf T}V r$ | | Mamba-2 | $O r=\overleftarrow Q rS r+ Q rK r^{\mathsf T}\odot\Gamma {C,r} V r$ | $S {r+1}=\lambda r 1:C S r+\overrightarrow K r^{\mathsf T}V r$ | | DeltaNet | $O r=Q rS r+ Q rK r^{\mathsf T}\odot M c Z r$ | $S {r+1}=S r+K r^{\mathsf T}Z r$ | | GDN | $O r=\overleftarrow Q rS r+ Q rK r^{\mathsf T}\odot\Gamma {C,r} Z r$ | $S {r+1}=\lambda r 1:C S r+\overrightarrow K r^{\mathsf T}Z r$ | Complete per-chunk algorithm For each chunk $r$, define the decay products and the correction gate: Gated Delta Product GDP Gated Delta Product GDP https://arxiv.org/abs/2502.10297 replaces GDN's single generalized Householder transition per token with an ordered product of $n h$ transitions. Each transition has its own key, value, and correction gate, all projected from the same input token. The transitions act along different learned key directions, increasing computation and expressiveness without changing the shape of $S t$. For substep $j$, define For example, with $n h=3$, the token update is The decay $a t$ is applied before the first substep, so the previous state decays only once. The query reads the state only after all three updates have finished. GDP as GDN over a virtual sequence GDP can be written as an ordinary GDN recurrence over a longer sequence. Treat its $n h$ substeps as consecutive positions in that sequence. We call the original positions real tokens and the expanded positions virtual steps . For $n h=3$, one real token expands into three virtual steps. Two real tokens therefore become six: | real token $t=1$ | real token $t=2$ | ||||| |---|---|---|---|---|---|---| | virtual step $m$ | 1 | 2 | 3 | 4 | 5 | 6 | | substep $ t,j $ | $ 1,1 $ | $ 1,2 $ | $ 1,3 $ | $ 2,1 $ | $ 2,2 $ | $ 2,3 $ | | $\tilde k m$ | $k {1,1}$ | $k {1,2}$ | $k {1,3}$ | $k {2,1}$ | $k {2,2}$ | $k {2,3}$ | | $\tilde v m$ | $v {1,1}$ | $v {1,2}$ | $v {1,3}$ | $v {2,1}$ | $v {2,2}$ | $v {2,3}$ | | $\tilde\beta m$ | $\beta {1,1}$ | $\beta {1,2}$ | $\beta {1,3}$ | $\beta {2,1}$ | $\beta {2,2}$ | $\beta {2,3}$ | | $\tilde a m$ | $a 1$ | 1 | 1 | $a 2$ | 1 | 1 | | $\tilde q m$ | 0 | 0 | $q 1$ | 0 | 0 | $q 2$ | | retained output | — | — | $o 1$ | — | — | $o 2$ | The decay appears at the first virtual step of each real token; the remaining steps use $1$. The query appears at the last virtual step; the earlier steps use zero queries because they do not produce outputs. To write this construction generally, flatten token $t$ and substep $j$ into Map the GDP inputs to this virtual sequence by setting Every virtual step then follows the GDN recurrence: Flattening preserves the update order. We can therefore run the GDN chunkwise algorithm on a virtual sequence of length $nn h$ and retain every $n h$-th output. Only the sequence becomes longer; the recurrent state keeps the same shape. Gated Linear Attention GLA Mamba-2 lets each token choose how much of the previous state survives, but its gate is scalar: The same $a t$ scales every row of $S {t-1}$. It can shorten or extend the lifetime of the memory as a whole, but it cannot preserve one channel while rapidly forgetting another. Gated Linear Attention GLA https://arxiv.org/abs/2312.06635 replaces the scalar gate with a vector and uses the recurrence The two gates scale the same state differently: GLA therefore gives each channel its own forgetting rate. Chunkwise form Multiplying diagonal decay matrices is equivalent to multiplying their diagonal vectors elementwise: We therefore replace Mamba-2's scalar prefix product with a vector prefix product. Within chunk $r$, define Here, $\bigodot$ denotes a repeated Hadamard elementwise product. After a write enters the state, later tokens decay each of its channels independently: Incoming state. The incoming state reaches local position $\ell$ through $\boldsymbol\lambda r 1:\ell $. Stack these vectors and absorb them into $Q r$: Local writes. A write at position $m$ reaches position $\ell$ through $\boldsymbol\lambda r m+1:\ell $. Its local score is therefore For the state update, absorb the decay from each write to the right boundary into its key: The complete chunkwise equations are Kimi Delta Attention KDA Kimi Delta Attention KDA https://arxiv.org/abs/2510.26692 adds GLA's channelwise decay to GDN's delta rule: KDA first decays the state channel by channel, then applies the delta correction. If $\boldsymbol a t=a t\boldsymbol 1$, it reduces exactly to GDN. Reuse GLA and GDN The GLA and GDN sections give most of the chunkwise algorithm. GLA gives the decayed matrices $\overleftarrow Q r$ and $\overrightarrow K r$, together with the local score matrix $A r$. GDN shows that the delta rule replaces the value rows $V r$ with correction rows $Z r$. Therefore, It remains to compute $Z r$. KDA reuses GDN's correction solve but replaces scalar decay with channelwise decay. In this solve, decay appears in two quantities: $L r$ and $\overleftarrow K r$. KDA keeps this structure and replaces the scalar decay products with channelwise products: | Quantity | GDN: scalar decay | KDA: channelwise decay | |---|---|---| | Decay from $i$ through $j$ | $\lambda r i:j =\prod {s=i}^{j}a {r,s}$ | $\boldsymbol\lambda r i:j =\bigodot {s=i}^{j}\boldsymbol a {r,s}$ | | Incoming prediction key | $\overleftarrow k {r,\ell}=\lambda r 1:\ell k {r,\ell}$ | $\overleftarrow k {r,\ell}=\boldsymbol\lambda r 1:\ell \odot k {r,\ell}$ | | $L r \ell,m $ for $m<\ell$ | $\lambda r m+1:\ell k {r,\ell}k {r,m}^{\mathsf T} $ | $k {r,\ell}\bigl \boldsymbol\lambda r m+1:\ell \odot k {r,m}\bigr ^{\mathsf T}$ | Using the new $L r$ and $\overleftarrow K r$, reuse GDN's triangular transform. Let $B r:=\operatorname{diag} \beta {r,1},\ldots,\beta {r,C} $. Then Complete per-chunk algorithm For each chunk $r$, define the channelwise decay products and correction gate: Stack the transformed rows into $\overleftarrow Q r$, $\overleftarrow K r$, and $\overrightarrow K r$. Define the two causal within-chunk matrices: Sources and further reading Related papers Transformers are RNNs https://arxiv.org/abs/2006.16236 , , https://arxiv.org/abs/2307.08621 RetNet , https://arxiv.org/abs/2312.06635 Gated Linear Attention GLA , https://arxiv.org/abs/2405.21060 Mamba-2 / SSD , https://arxiv.org/abs/2406.06484 DeltaNet , https://arxiv.org/abs/2412.06464 Gated DeltaNet GDN , and https://arxiv.org/abs/2502.10297 DeltaProduct GDP Kimi Linear / Kimi Delta Attention KDA