In July 2026, Moonshot released Kimi K3, a 2T-parameter model with a 1M-token context window. K3 uses Moonshot's linear-attention variant, Kimi Delta Attention (KDA). Moonshot reports that it trails only Claude Fable 5 and GPT-5.6 Sol across its evaluation suite.
NVIDIA had already made a similar choice in Nemotron 3, which combines softmax attention with Mamba-2 recurrent layers. KDA and Mamba-2 use different recurrences. Both offer linear-time sequence mixing during training and fixed-state decoding, although this efficiency comes with some loss of expressiveness.
In this post, we visualize the path from self-attention to the linear-attention variants used in recent models.
Self-attention recap #
Given hidden states $H$, three learned projections produce queries, keys, and values:
where $Q,K\in\mathbb R^{n\times d_k}$ and $V,O\in\mathbb R^{n\times d_v}$, with $n$ denoting the sequence length.
Each query $q_t$ represents a linear functional on the key space:
Evaluating it at $k_j$ produces the scalar score $\langle q_t,k_j\rangle$. Softmax normalizes these scores into weights, and $o_t$ is the weighted sum of the corresponding values.
Self-attention is quadratic #
The attention operation uses two matrix multiplications over the sequence:
Masking and softmax add $\mathcal O(n^2)$ work, so the attention operation costs
which is quadratic in the sequence length.
A direct implementation stores the $n\times n$ score matrix. FlashAttention avoids materializing this matrix in off-chip memory, but the computation remains quadratic.
Optional: Attention vs. FFN #
For model width $d$ and FFN expansion factor $e$:
For $e=4$, the full attention layer (attention operation + input/output projections) costs more than the FFN when $n>2d$; the attention operation alone does so when $n>4d$.
Naïve linear attention #
The quadratic cost comes from multiplying $Q$ by $K^{\mathsf T}$ first, which produces an $n\times n$ score matrix. If we remove softmax, matrix multiplication is associative, so we can change the order of computation:
The left-hand order compares every query with every key before mixing the values. The right-hand order first summarizes all key–value pairs in the $d\times d$ matrix $K^{\mathsf T}V$, then applies each query to that summary. It never constructs an $n\times n$ matrix.
The two matrix multiplications now cost
Together, they require approximately $4nd^2=\mathcal O(nd^2)$ work, which is linear rather than quadratic in the sequence length.
This computation is not causal: the shared summary $K^{\mathsf T}V$ contains key–value pairs from the entire sequence, so every query can read the future. The causal version is
At first glance, the causal mask appears to prevent direct reassociation of the full matrix product. However, row $t$ of $O$ depends only on $q_t$, $K_{1:t}$, and $V_{1:t}$, as shown below. (Hover a row of $O$ to trace its prefix computation.)
Once the masked columns are removed, row $t$ becomes an ordinary matrix product over a prefix:
We call the product of the key and value prefixes the state $S_t$:
$S_t$ is a fixed-size linear map from key space to value space. Moving from one token to the next adds exactly one outer product, so we can build this state recurrently instead of recomputing the entire prefix:
Each recurrent step costs $\mathcal O(d^2)$, so processing $n$ tokens costs $\mathcal O(nd^2)$. The cost is linear in the sequence length, hence the name linear attention.
Optional: Linear attention as an RNN #
This recurrence defines an RNN with a matrix-valued hidden state $S_t$. Each key–value pair updates $S_t$. The query $q_t$ reads $S_t$ to produce $o_t$. This recurrent view is the idea behind Transformers are RNNs: Fast Autoregressive Transformers with Linear Attention.
The recurrent form reduces the arithmetic complexity, but it uses GPUs inefficiently:
Large activation memory. A direct autograd implementation saves every $d\times d$ state $S_t$ for the backward pass. The $n$ saved states require $\mathcal O(nd^2)$ memory.Sequential token updates. Computing $S_t$ requires $S_{t-1}$. This dependency prevents the GPU from updating different tokens in parallel.Poor Tensor Core utilization. Tensor Cores accelerate tiled matrix multiply-accumulate operations. The outer product $k_t^{\mathsf T}v_t$ has shape $(d\times1)(1\times d)$, so its reduction dimension is only one. This shape does not fill the Tensor Core tiles along the reduction dimension.
Chunkwise algorithm #
To solve these problems, we introduce the chunkwise algorithm. It uses the parallel form $\bigl(QK^{\mathsf T}\odot M\bigr)V$ within each chunk and the recurrent form $S_{\mathrm{out}}=S_{\mathrm{in}}+K^{\mathsf T}V$ between chunks.
Assume $C$ divides $n$. Split the sequence into $n/C$ chunks of $C$ consecutive tokens. Let $Q_r,K_r,V_r\in\mathbb R^{C\times d}$ denote chunk $r$.
Block the causal matrix
Let $A=QK^{\mathsf T}\odot M$. Partition $A$ into $C\times C$ blocks and partition $V$ and $O$ into $C\times d$ blocks. Let $A_{r,s}$ denote block $(r,s)$ of $A$. In block row $r$, blocks to the left contain earlier chunks, the diagonal block contains the current chunk, and blocks to the right contain future tokens. A n×n in C×C blocks
The output for chunk $r$ splits into history and local contributions:
The local block keeps a $C\times C$ causal mask because tokens in the chunk have different prefix lengths. The history blocks need no mask because every token in chunk $r$ may read every token in an earlier chunk.
Compress the history
Let $K_{<r}$ and $V_{<r}$ contain all chunks before chunk $r$. Associativity compresses the dense history: $S_r$ contains all key–value updates before chunk $r$. Let $M_c$ denote the $C\times C$ causal mask inside the chunk. The chunk output is
Update the state
After computing $O_r$, add the current chunk to the history:
The matrix product $K_r^{\mathsf T}V_r$ batches $C$ rank-one updates into a dense GEMM. Its reduction dimension is $C$ rather than $1$, so it uses Tensor Cores efficiently.
For the recurrent state, autograd now saves one boundary state per chunk. These $n/C$ states require $\mathcal O((n/C)d^2)$ memory. Chunk boundaries still form a sequential chain, but the chain now contains $n/C$ steps instead of $n$.
Cost
Each chunk requires
Across $n/C$ chunks, the total cost is
Larger chunks increase the local $nCd$ term but shorten the sequential state chain. $C=1$ recovers the recurrent form, while $C=n$ recovers the full parallel form. Every valid $C$ provides exact results. FLA uses $C=64$, while FlashKDA uses $C=16$.
RetNet: scalar decay #
In Naïve linear attention, each outer-product write $k_t^{\mathsf T}v_t$ enters the state with coefficient one and never decays. New and old writes therefore receive the same age-independent coefficient. RetNet introduces a recency bias by shrinking the existing state before adding the current write:
$\gamma$ is a fixed scalar, so it scales the entire state uniformly. The figure follows one key–value write through the recurrence:
A write from $s$ steps earlier has weight $\gamma^s$. The weight decreases exponentially with age.
Parallel form
To compute the recurrence in parallel, note that a write at position $j$ undergoes $t-j$ decay steps before query $q_t$ reads it. Its coefficient is therefore $\gamma^{t-j}$. Collect these coefficients in a decayed causal matrix: The diagonal is one because the current write has not decayed. Moving one column to the left adds one decay step.
The parallel form replaces the binary causal mask $M$ with $\Gamma$:
Chunkwise form
The chunkwise form again separates incoming history from interactions inside the chunk. Let $S_r$ denote the state immediately before chunk $r$, and let $\ell=1,\ldots,C$ index positions in the current chunk.
Decay incoming history
Before $q_{r,\ell}$ (the query at local position $\ell$ in chunk $r$) reads $S_r$, the state crosses $\ell$ decay steps. Define the boundary-to-query factors
The first illustration applies one entry of $\lambda$ to each row of $Q_rS_r$. (Hover a row of $Q_r$ to trace its decay factor and the resulting history row.)
We can move the row scale to $Q_r$ and define the decayed query:
The left arrow denotes left-boundary decay: row $\ell$ includes the factor $\gamma^\ell$ from the chunk boundary to $q_{r,\ell}$.
This gives the history term
Compute the local block
Partitioning the full parallel computation into $C\times C$ blocks exposes the same history and local split:
T⊙ Γ)decayed scores · n×n → C×C blocks
local
history
A write at local position $m$ crosses $\ell-m$ decay steps before $q_{r,\ell}$ reads it. The local decay matrix is
The local term is
Update the state
We now have the chunk output $O_r$. Next, we construct the state passed to the next chunk. The incoming state crosses all $C$ steps, while a write at local position $\ell$ crosses only $C-\ell$ steps. Define the write-to-boundary factors Each token contributes one decayed outer-product write:
For $C=2$, the first equality expands the shared chunk dimension and applies $ρ_ℓ$ to deposit $ℓ$. The two partial updates then sum to $\Delta S_r$. Hover a scale or partial update to trace the corresponding key column and value row. We can then absorb these two scale factors into the corresponding rows of $K_r$:
The right arrow denotes right-boundary decay: key row $\ell$ includes the factor $\gamma^{C-\ell}$ from its position to the chunk's right boundary.
Complete chunkwise form
Combining the history and local terms gives the complete chunkwise equations:
The arrows identify the relevant boundary. Queries absorb decay from the left boundary, while keys absorb decay toward the right boundary. The incoming history state decays from the left boundary to each query position; each key–value write made inside the chunk decays from its position to the right boundary. This exact regrouping keeps the same $\mathcal O(nCd+nd^2)$ asymptotic cost as the Naïve chunkwise algorithm.
Mamba-2: input-dependent scalar decay #
RetNet applies the same decay $\gamma$ at every position. Mamba-2 replaces it with a token-dependent scalar $a_t$. Its recurrent form is
Each token supplies one scalar for the entire state. A value near one preserves history; a value near zero nearly clears it.
Optional: Constructing the decay gate #
The recurrence assumes $0<a_t<1$. During training, the model must learn $a_t$ while preserving this constraint. Mamba-2 constructs the gate by discretizing continuous-time exponential decay:
Over one token interval $\Delta_t$, the exact solution is
First, Mamba-2 learns a positive decay rate. It stores an unconstrained log-rate $A_{\log}$ and exponentiates it:
The rate $\lambda$ specifies the decay per unit time. Discretization converts the continuous system into one transition per token. Mamba-2 computes an input-dependent step size $\Delta_t$ for the transition at token $t$, so different tokens advance the continuous system by different amounts. A token with a larger $\Delta_t$ advances the system farther and produces stronger decay; a token with a smaller $\Delta_t$ produces weaker decay. From the token representation $h_t$, a learned projection $W_\Delta$ produces an unconstrained step $\widetilde\Delta_t$. The model adds a learned bias $b_\Delta$ and applies softplus to make the step positive:
Combining the positive rate and interval gives
Parallel form
The outer-product write $k_j^{\mathsf T}v_j$ enters the state at position $j$. To reach $S_t$, subsequent transitions multiply it by $a_{j+1},a_{j+2},\ldots,a_t$. Its survival coefficient is
Collect these survival coefficients in the decay matrix $\Gamma$. Entry $\Gamma[t,j]$ scales the score between $q_t$ and $k_j$. (Hover an entry of $\Gamma$ to trace the corresponding query, key, value, and output.)
For an inclusive interval, define The decay matrix becomes
The parallel equation keeps the same form as RetNet:
Chunkwise form
The chunkwise derivation follows RetNet. Each fixed power of $\gamma$ is replaced by a product of token-dependent gates.
Within chunk $r$, let $a_{r,\ell}$ denote the gate at local position $\ell$, and define
Decay incoming history
Before $q_{r,\ell}$ reads the incoming state $S_r$, the recurrence multiplies $S_r$ by $\lambda_r[1:\ell]$. As in RetNet, absorb this scalar into the query row:
The left arrow again denotes decay from the chunk's left boundary to each query position.
Compute the local block
The write $k_{r,m}^{\mathsf T}v_{r,m}$ is multiplied by $\lambda_r[m+1:\ell]$ before it contributes to the output at local position $\ell$. Therefore, The local output is
Update the state
We now have $O_r$. To construct the state for the next chunk, multiply the incoming state by $\lambda_r[1:C]$. The write at local position $\ell$ is multiplied only by the subsequent gates, whose product is $\lambda_r[\ell+1:C]$. Absorb this suffix product into its key: (Hover an element of $\operatorname{diag}(\lambda_r[\ell+1:C])$ to trace the corresponding key column and value row.)
The right arrow denotes decay from each write position to the chunk's right boundary. The state update is
Complete chunkwise form
Combining the output and state equations gives
Mamba-2 therefore keeps RetNet's scalar state transition but lets each token choose how much history survives. The gate remains scalar, so it scales every entry of the state equally.
Optional: Discretization convention #
Discretization affects both the decay and the new write. Mamba-2 also multiplies the write by the step size $\Delta_t$, so the literal recurrence is
To keep the same notation as the preceding sections, define
All equations above use this rescaled $v_t$.
DeltaNet: correcting memory collisions #
Mamba-2 decays the entire history at once. Sometimes we instead want to correct the association addressed by the current key without uniformly decaying the entire memory. DeltaNet provides this more selective update.
Why additive writes collide
Treat $S$ as a key–value associative memory. For each association $(k_i,v_i)$, we would like to retrieve $v_i$ by reading the memory with $k_i$:
However, for a unit-norm key $k_i$, the read contains
The first term is the intended value. The other terms come from keys that overlap with $k_i$. Orthogonal keys eliminate this interference, but learned keys generally are not mutually orthogonal. A raw outer-product write can therefore collide with associations already stored in $S$.
DeltaNet first asks what value the current state associates with $k_t$:
The residual $v_t-\widehat v_t$ points from this prediction toward the desired value $v_t$. DeltaNet writes a step in that direction at $k_t$:
An intuitive example
Consider a simplified memory with two associations. The labels below stand for key and value vectors. Assume the Alice and Bob keys are orthogonal and unit norm, so a write at Bob's key does not change the value read at Alice's key.
Alice at school
Bob at home
Bob at the office
Reading the state with Bob's key first returns “at home.” The target is “at the office,” so DeltaNet writes the difference between those two value vectors at Bob's key. Reading Bob after the update gives
If $\beta=1$, the new value replaces the old prediction exactly. A smaller $\beta$ makes a partial correction. More generally, when $\lVert k_t\rVert_2=1$,
The unit-norm condition matters. Without it, reading the update back with $k_t$ introduces an additional factor $\lVert k_t\rVert_2^2$.
Optional: Test-time training (TTT) interpretation #
Another interpretation treats $S$ as a small linear model that maps keys to values. The pair $(k_t,v_t)$ supplies one online training example with squared-error loss
Its gradient is
One gradient step from $S_{t-1}$ with learning rate $\beta_t$ gives
which is exactly the DeltaNet update. This view connects the delta rule to Learning to (Learn at Test Time): the forward pass updates the recurrent state as a fast-weight model.
The same view also recovers the preceding recurrences. With a unit learning rate:
| Method | Online loss $\mathcal L_t(S)$ | State update |
|---|---|---|
| Naïve | $-\langle k_tS,v_t\rangle$ | $S_t=S_{t-1}+k_t^{\mathsf T}v_t$ |
| RetNet | $-\langle k_tS,v_t\rangle+\dfrac{1-\gamma}{2}\lVert S\rVert_F^2$ | $S_t=\gamma S_{t-1}+k_t^{\mathsf T}v_t$ |
| Mamba-2 | $-\langle k_tS,v_t\rangle+\dfrac{1-a_t}{2}\lVert S\rVert_F^2$ | $S_t=a_tS_{t-1}+k_t^{\mathsf T}v_t$ |
From the TTT view, Naïve linear attention has an unbounded objective: increasing $S$ along $k_t^{\mathsf T}v_t$ can drive the linear loss toward $-\infty$. RetNet adds a fixed $L_2$ penalty on the state, while Mamba-2 makes the strength of that penalty token-dependent through $a_t$.
The example explains one token update. We now derive the recurrent and full-sequence matrix forms.
Recurrent form
Define the scaled correction
The complete token recurrence is
Once $u_t$ is known, the state has the same additive form as Naïve linear attention:
Unlike the raw values $v_t$, the corrections $u_t$ are causally coupled: each $u_t$ depends on earlier corrections through $S_{t-1}$. We therefore cannot form all rows of $U$ independently as in Naïve linear attention. We first collect this dependency into a triangular system and solve for $U$; only then can we reuse the linear-attention readout.
The triangular correction system
Substitute
into the prediction. Then
For the first three tokens, Only earlier corrections appear on the right. Define the strictly lower-triangular matrix
Let $M$ be the inclusive causal mask and $M^-:=M-I$ its strictly lower-triangular part. Then
Stack $u_t$ into $U\in\mathbb R^{n\times d}$ and define $B:=\operatorname{diag}(\beta)$. The row recurrences become (as you may verify):
Equivalently,
Optional: Solving for U with forward substitution #
We do not compute $(I+BL)^{-1}$ explicitly. Forming a dense $n\times n$ inverse costs $\mathcal O(n^3)$. Set $A:=I+BL$ and $R:=BV$. Since $A$ is unit lower triangular, forward substitution solves
one row at a time. At row $t$, the earlier rows of $U$ are already known, and $u_t$ is the only new row. (Hover a row to follow the known inputs and current unknown.)
For $U\in\mathbb R^{n\times d}$, row $t$ costs $\mathcal O(td)$ and the full solve costs $\mathcal O(n^2d)$.
Parallel output
After solving for $U$, the state is a sum of key–correction outer products. The causal output therefore reuses the Naïve linear-attention equation with $U$ in place of $V$:
Chunkwise algorithm
The full $n$-token triangular system still requires quadratic work. Divide the sequence into chunks of $C$ tokens instead. Each chunk starts from its incoming boundary state. Local dependencies then fit in a $C\times C$ system, while a $d\times d$ state carries history between chunks. This regrouping is exact.
One chunk as an affine recurrence
Let $r$ index chunks and $\ell\in{1,\ldots,C}$ index tokens in the current chunk. $S_r$ is the state before the chunk, and $S_{r+1}$ is the state after it. Rewrite the DeltaNet update as
For example, a chunk with $C=2$ contains two local transitions:
Use $\Lambda_r[i:j]$ for the ordered transition product over local positions $i$ through $j$ in chunk $r$. Applying the lemma to the incoming state $S_r$ gives
The first term carries the incoming state across the chunk. $H_r$ is the within-chunk convolution sum of local writes. Computing either expression directly would require many dense $d\times d$ transition products.
Compact the transition product with WY
For DeltaNet, the two vectors in each DPLR correction are $k_{r,\ell}$ and $\beta_{r,\ell}k_{r,\ell}$. We absorb $\beta_{r,\ell}$ into the triangular factor below. WY then gives To construct $W_r$, maintain the partial form
The first three rows of $W_r$ follow by substituting one transition at a time:
The general row is
Let $M_c$ be the inclusive $C\times C$ causal mask, $M_c^-:=M_c-I$, and As in the full-sequence parallel form, stacking these row recurrences produces:
Compact the local writes
The local convolution sum $H_r$ is the state produced by the tokens in chunk $r$ when the incoming state is zero. The full-sequence parallel form already showed that DeltaNet writes its state as $K^{\mathsf T}U$. Applying the same result within the chunk gives
The first three rows of $U_r$ are (as you may verify)
The general row is
This is exactly the full-sequence correction recurrence, applied only within chunk $r$. Stacking the rows gives:
Optional: Use UT to share the triangular transform #
Solving for $W_r$ and $U_r$ separately would repeat the same lower-triangular solve. UT lets us solve once for a shared transform, then obtain both matrices with dense matrix multiplications.
UT representation. The WY form leaves $W$ implicit. For the DPLR product above, let $V$ stack the right factors $v_1,\ldots,v_m$. Because row $i$ of $W$ depends only on $v_1,\ldots,v_i$, we can write
for a lower-triangular $T\in\mathbb R^{m\times m}$. The WY representation then becomes This equivalent form is the UT representation. It exposes the triangular transform from the original right factors $V$ to their accumulated form $W$.
The derivations of $W_r$ and $U_r$ produced
The coefficient matrix is identical; only the right-hand side changes. Following UT, define $T_r\in\mathbb R^{C\times C}$ by
Right multiplication by $K_r$ or $V_r$ then gives
The compact transition product and convolution sum are therefore
Correct the incoming state
Substitute the compact forms of $\Lambda_r[1:C]$ and $H_r$ into the unrolled recurrence:
Define the actual correction rows as
$U_r$ contains the corrections obtained with an empty incoming state. $W_rS_r$ is the transformed contribution of the predictions from $S_r$. Their difference gives the corrections produced by the actual chunk:
The update now has the same additive form as Naïve linear attention. We can therefore reuse its chunkwise output formula with $Z_r$ in place of $V_r$:
Complete per-chunk algorithm #
For each chunk $r$, The local quantities $L_r,T_r,W_r,$ and $U_r$ can be computed for all chunks in parallel. The boundary pass then threads $S_r$ through the chunks to form $Z_r$, $O_r$, and $S_{r+1}$.
## Gated DeltaNet (GDN)
[Gated DeltaNet (GDN)](https://arxiv.org/abs/2412.06464) extends DeltaNet by adding a token-dependent scalar decay to the state transition:
GDN adds scalar decay to DeltaNet in the same way that Mamba-2 adds scalar decay to Naïve linear attention. DeltaNet also has the same chunkwise form as Naïve linear attention after replacing $V_r$ with $Z_r$.
| Method | Chunkwise readout | Chunkwise state update |
|---|---|---|
| Naïve | $O_r=Q_rS_r+(Q_rK_r^{\mathsf T}\odot M_c)V_r$ | $S_{r+1}=S_r+K_r^{\mathsf T}V_r$ |
| Mamba-2 | $O_r=\overleftarrow Q_rS_r+(Q_rK_r^{\mathsf T}\odot\Gamma_{C,r})V_r$ | $S_{r+1}=\lambda_r[1:C]S_r+\overrightarrow K_r^{\mathsf T}V_r$ |
| DeltaNet | $O_r=Q_rS_r+(Q_rK_r^{\mathsf T}\odot M_c)Z_r$ | $S_{r+1}=S_r+K_r^{\mathsf T}Z_r$ |
| GDN | $O_r=\overleftarrow Q_rS_r+(Q_rK_r^{\mathsf T}\odot\Gamma_{C,r})Z_r$ | $S_{r+1}=\lambda_r[1:C]S_r+\overrightarrow K_r^{\mathsf T}Z_r$ |
Complete per-chunk algorithm #
For each chunk $r$, define the decay products and the correction gate:
## Gated Delta Product (GDP)
Gated Delta Product (GDP) replaces GDN's single generalized Householder transition per token with an ordered product of $n_h$ transitions.
Each transition has its own key, value, and correction gate, all projected from the same input token. The transitions act along different learned key directions, increasing computation and expressiveness without changing the shape of $S_t$.
For substep $j$, define
For example, with $n_h=3$, the token update is
The decay $a_t$ is applied before the first substep, so the previous state decays only once. The query reads the state only after all three updates have finished.
GDP as GDN over a virtual sequence
GDP can be written as an ordinary GDN recurrence over a longer sequence. Treat its $n_h$ substeps as consecutive positions in that sequence. We call the original positions real tokens and the expanded positions virtual steps.
For $n_h=3$, one real token expands into three virtual steps. Two real tokens therefore become six:
| real token $t=1$ | real token $t=2$ | |||||
|---|---|---|---|---|---|---|
| virtual step $m$ | 1 | 2 | 3 | 4 | 5 | 6 |
| substep $(t,j)$ | $(1,1)$ | $(1,2)$ | $(1,3)$ | $(2,1)$ | $(2,2)$ | $(2,3)$ |
| $\tilde k_m$ | $k_{1,1}$ | $k_{1,2}$ | $k_{1,3}$ | $k_{2,1}$ | $k_{2,2}$ | $k_{2,3}$ |
| $\tilde v_m$ | $v_{1,1}$ | $v_{1,2}$ | $v_{1,3}$ | $v_{2,1}$ | $v_{2,2}$ | $v_{2,3}$ |
| $\tilde\beta_m$ | $\beta_{1,1}$ | $\beta_{1,2}$ | $\beta_{1,3}$ | $\beta_{2,1}$ | $\beta_{2,2}$ | $\beta_{2,3}$ |
| $\tilde a_m$ | $a_1$ | 1 | 1 | $a_2$ | 1 | 1 | | $\tilde q_m$ | 0 | 0 | $q_1$ | 0 | 0 | $q_2$ | | retained output | — | — | $o_1$ | — | — | $o_2$ |
The decay appears at the first virtual step of each real token; the remaining steps use $1$. The query appears at the last virtual step; the earlier steps use zero queries because they do not produce outputs.
To write this construction generally, flatten token $t$ and substep $j$ into
Map the GDP inputs to this virtual sequence by setting
Every virtual step then follows the GDN recurrence:
Flattening preserves the update order. We can therefore run the GDN chunkwise algorithm on a virtual sequence of length $nn_h$ and retain every $n_h$-th output. Only the sequence becomes longer; the recurrent state keeps the same shape.
Gated Linear Attention (GLA) #
Mamba-2 lets each token choose how much of the previous state survives, but its gate is scalar:
The same $a_t$ scales every row of $S_{t-1}$. It can shorten or extend the lifetime of the memory as a whole, but it cannot preserve one channel while rapidly forgetting another.
Gated Linear Attention (GLA) replaces the scalar gate with a vector and uses the recurrence
The two gates scale the same state differently:
GLA therefore gives each channel its own forgetting rate.
Chunkwise form
Multiplying diagonal decay matrices is equivalent to multiplying their diagonal vectors elementwise:
We therefore replace Mamba-2's scalar prefix product with a vector prefix product. Within chunk $r$, define
Here, $\bigodot$ denotes a repeated Hadamard (elementwise) product.
After a write enters the state, later tokens decay each of its channels independently:
Incoming state. The incoming state reaches local position $\ell$ through $\boldsymbol\lambda_r[1:\ell]$. Stack these vectors and absorb them into $Q_r$:
Local writes. A write at position $m$ reaches position $\ell$ through $\boldsymbol\lambda_r[m+1:\ell]$. Its local score is therefore
For the state update, absorb the decay from each write to the right boundary into its key: The complete chunkwise equations are
## Kimi Delta Attention (KDA)
[Kimi Delta Attention (KDA)](https://arxiv.org/abs/2510.26692) adds GLA's channelwise decay to GDN's delta rule:
KDA first decays the state channel by channel, then applies the delta correction. If $\boldsymbol a_t=a_t\boldsymbol 1$, it reduces exactly to GDN.
Reuse GLA and GDN
The GLA and GDN sections give most of the chunkwise algorithm. GLA gives the decayed matrices $\overleftarrow Q_r$ and $\overrightarrow K_r$, together with the local score matrix $A_r$. GDN shows that the delta rule replaces the value rows $V_r$ with correction rows $Z_r$. Therefore,
It remains to compute $Z_r$. KDA reuses GDN's correction solve but replaces scalar decay with channelwise decay. In this solve, decay appears in two quantities: $L_r$ and $\overleftarrow K_r$.
KDA keeps this structure and replaces the scalar decay products with channelwise products:
| Quantity | GDN: scalar decay | KDA: channelwise decay |
|---|---|---|
| Decay from $i$ through $j$ | $\lambda_r[i:j]=\prod_{s=i}^{j}a_{r,s}$ | $\boldsymbol\lambda_r[i:j]=\bigodot_{s=i}^{j}\boldsymbol a_{r,s}$ |
| Incoming prediction key | $\overleftarrow k_{r,\ell}=\lambda_r[1:\ell]k_{r,\ell}$ | $\overleftarrow k_{r,\ell}=\boldsymbol\lambda_r[1:\ell]\odot k_{r,\ell}$ |
| $L_r[\ell,m]$ for $m<\ell$ | $\lambda_r[m+1:\ell](k_{r,\ell}k_{r,m}^{\mathsf T})$ | $k_{r,\ell}\bigl(\boldsymbol\lambda_r[m+1:\ell]\odot k_{r,m}\bigr)^{\mathsf T}$ |
Using the new $L_r$ and $\overleftarrow K_r$, reuse GDN's triangular transform. Let $B_r:=\operatorname{diag}(\beta_{r,1},\ldots,\beta_{r,C})$. Then
Complete per-chunk algorithm #
For each chunk $r$, define the channelwise decay products and correction gate: Stack the transformed rows into $\overleftarrow Q_r$, $\overleftarrow K_r$, and $\overrightarrow K_r$. Define the two causal within-chunk matrices:
Sources and further reading #
Related papers
[ Transformers are RNNs](https://arxiv.org/abs/2006.16236),
[,](https://arxiv.org/abs/2307.08621)
*RetNet*[,](https://arxiv.org/abs/2312.06635)
*Gated Linear Attention (GLA)*[,](https://arxiv.org/abs/2405.21060)
*Mamba-2 / SSD*[,](https://arxiv.org/abs/2406.06484)
*DeltaNet*[,](https://arxiv.org/abs/2412.06464)
*Gated DeltaNet (GDN)*[, and](https://arxiv.org/abs/2502.10297)
*DeltaProduct (GDP)*
Kimi Linear / Kimi Delta Attention (KDA)