# Kimi Linear: How Moonshot AI Built a Hybrid Attention Architecture That Beats Full Attention > Source: > Published: 2026-09-25 16:05:49+00:00 The standard Transformer attention mechanism has a well-known problem: it scales quadratically with sequence length. As context windows push toward one million tokens, the KV cache alone can consume tens of gigabytes of memory, and decoding slows to a crawl. Researchers have proposed linear attention as a fix for years, but linear attention models have historically underperformed full attention on standard benchmarks — making the trade-off feel like a step backward. Moonshot AI's [Kimi Linear](https://arxiv.org/abs/2510.26692) paper, released alongside open-source model checkpoints and kernels, argues that this trade-off is no longer necessary. By introducing a new attention module called Kimi Delta Attention (KDA) and combining it with full attention in a carefully tuned hybrid ratio, the architecture achieves up to 75% KV cache reduction and 6.3× faster decoding at one million token context lengths — while matching or exceeding full-attention baselines on both short- and long-context benchmarks. Standard self-attention computes a weighted sum over all previous tokens, which is expressive but expensive. Linear attention reformulates this as a recurrent state update: instead of attending to every past token, the model maintains a fixed-size memory matrix that gets updated incrementally. This brings complexity down from O(n²) to O(n), but the fixed-size memory creates a bottleneck — the model can only "remember" what fits in that matrix, and older information gets overwritten. Previous linear attention variants like [GLA and Gated DeltaNet](https://arxiv.org/abs/2407.08608) improved on this by adding gating mechanisms, but they applied gating at the level of entire attention heads. This coarse granularity limits how selectively the model can manage its memory across different feature dimensions. KDA's key innovation is moving from head-level gating to **channel-wise gating**. In the KDA update rule, each feature dimension gets its own decay scalar (α_t), allowing individual channels to evolve independently. The update takes the form: ``` S_t = (I − β_t k_t k_t^T) · Diag(α_t) · S_{t-1} + β_t k_t v_t^T ``` This combines three operations: a per-channel decay (Diag(α_t)), a rank-1 forget operation that removes the projection of the previous state onto the current key, and a rank-1 write that adds new information. The result is a more expressive memory update that can selectively retain or discard information at a finer granularity than head-level gating allows. To make this computationally efficient on GPUs, the authors use a **Diagonal-Plus-Low-Rank (DPLR)** structure for the transition matrices. Sequences are split into chunks of 256 tokens, and a WY representation compresses multiple rank-1 updates into a single form that maps cleanly to triangular solves on Tensor Cores. This is what makes the architecture practical rather than just theoretically appealing. Rather than replacing full attention entirely, Kimi Linear uses a **3:1 hybrid ratio**: three KDA layers for every one Multi-Head Latent Attention (MLA) layer. MLA is the same attention mechanism used in [DeepSeek models](https://arxiv.org/abs/2501.12599) — it compresses keys and values into a low-dimensional latent space before projecting them back, reducing KV cache size while preserving expressiveness. The intuition behind the hybrid design is that linear recurrence handles local reasoning and memory compression efficiently, while the periodic full-attention layers provide global coherence that pure recurrent models struggle to maintain. The 3:1 ratio was validated empirically across multiple scales. One notable design choice: Kimi Linear uses **NoPE (No Positional Encoding)**. Instead of adding RoPE or sinusoidal embeddings, positional awareness emerges naturally from the decay dynamics within KDA. As tokens age, the channel-wise decay gates gradually reduce their influence on the memory state — a form of implicit positional encoding that avoids the distortions that can arise when bolting on external positional schemes. The released Kimi Linear model has 48B total parameters with 3B activated per token (a sparse MoE configuration). On benchmarks: The [GitHub repository](https://github.com/MoonshotAI/Kimi-Linear) notes that [Kimi K3](https://platform.kimi.ai/docs/guide/kimi-k3-quickstart) — Moonshot's flagship model — uses this architecture at scale, with 69 KDA layers and 24 Gated MLA layers. That's a real-world validation of the approach at 2.8 trillion total parameters. Moonshot AI released: `KimiLinearForCausalLM` class The vLLM integration requires some care: because KDA maintains a recurrent state rather than a KV cache, prefix caching works differently. The [LMCache documentation](https://docs.lmcache.ai/recipes/kimi_linear.html) covers the specifics of managing the recurrent state cache in production. For teams running inference at long context lengths, the efficiency gains here are substantial. A 75% reduction in KV cache memory means you can serve roughly 4× more concurrent long-context requests on the same hardware. The 6× decoding speedup at 1M tokens addresses one of the most painful bottlenecks in production long-context deployments. The architecture is also a meaningful step toward making hybrid linear-full attention a practical default rather than a research curiosity. The key insight — that channel-wise gating gives linear attention enough expressiveness to compete with full attention — is likely to influence future architecture work. Several recent models (including Kimi K3) have already adopted this design at scale, which suggests the approach is robust enough to survive contact with real training runs. The main caveat is that the released checkpoint is a 48B/3B-activated sparse model, which requires specific infrastructure (vLLM with FLA kernels, LMCache for state management). Teams running dense models or using standard inference stacks will need to adapt before they can take advantage of the architecture directly. Kimi Linear makes a credible case that the long-standing performance gap between linear and full attention can be closed with the right gating mechanism and hybrid design. The combination of channel-wise KDA gating, DPLR-based chunkwise computation, and a 3:1 hybrid ratio with MLA produces an architecture that is both more efficient and more accurate than full attention at long context lengths. With open-source kernels, vLLM support, and a flagship model already running on this architecture, the approach is ready to be evaluated in real deployments.