# Kimi K3's 1M context runs mostly on linear attention

> Source: <https://usewire.io/blog/kimi-k3-1m-context-linear-attention/>
> Published: 2026-07-17 17:00:00+00:00

[ AI Agent Context Engineering ](/blog/why-ai-agents-forget-mid-task/)

### Why AI agents forget mid-task (and how to fix it)

Key takeaway

Kimi K3's 1M-token context window is served mostly by Kimi Delta Attention, a linear attention design that compresses history into a fixed-size state instead of attending over every token. In the published Kimi Linear architecture, three of every four layers use this fixed-state mechanism, which is what makes million-token decoding up to 6.3x faster and cuts KV cache by up to 75%. The trade is that most of the model's layers are doing lossy compression the developer does not control, so context that has to be right should be offloaded to a lossless store outside the window and retrieved on demand, not relied on to survive in it. The case for that discipline gets stronger as long context gets cheaper, not weaker.

Kimi K3’s headline feature, a 1M-token context window with decoding up to 6.3x faster than its predecessor, is possible because most of the model does not do full attention. K3 is built on Kimi Delta Attention (KDA), a hybrid linear attention design in which the majority of layers compress everything they have read into a fixed-size memory state instead of attending over every cached token. In the published architecture this design descends from, that majority is three layers out of every four.

That is the interesting part of the July 16 release, more than the parameter count. Moonshot made long context dramatically cheaper by making most of the network’s memory lossy, and the model decides what to keep. For teams doing [context engineering](/glossary/context-engineering), that changes the economics of the window without changing any of the reasons to curate what goes into it.

Kimi K3 is the largest open-weights release to date: a 2.8-trillion-parameter mixture-of-experts model that activates 16 of 896 experts per token, takes image input natively, and serves a 1M-token [context window](/glossary/context-window), with full weights promised by July 27. Moonshot released it on July 16 with API pricing of $3.00 per million input tokens ($0.30 cached) and $15.00 per million output tokens, which Simon Willison notes makes it the most expensive model from a Chinese lab to date; K2.6 was priced at $0.95 and $4.

The release benchmarks position K3 just below the closed frontier and above everything open:

| Model | Artificial Analysis Intelligence Index |
|---|---|
| Claude Fable 5 | 60 |
| GPT-5.6 Sol | 59 |
| Kimi K3 | 57 |
| Claude Opus 4.8 | 56 |

The agentic numbers are the standouts: 91.2% on BrowseComp, the best published score at release, 84.2% on MCP Atlas, and 56.0% on Humanity’s Last Exam with tools. K3 also uses 21% fewer output tokens than K2.6 to get there, and ships with a single reasoning effort level, max.

Architecturally, three named components carry the release: Kimi Delta Attention for the sequence mixing, Stable LatentMoE for expert routing, and Attention Residuals (AttnRes), a depth-wise mechanism that selectively retrieves representations from earlier layers rather than accumulating them uniformly, which Moonshot credits with roughly 25% higher training efficiency at under 2% added cost. The first of the three is the one that matters for anyone deciding how much context to send.

KDA is linear attention: instead of comparing each new token against a growing cache of every previous key and value, it maintains a fixed-size memory state that gets updated once per token. Full attention’s cost structure is what makes million-token contexts painful. The KV cache grows linearly with input length, and every generated token pays attention over all of it, so both memory and per-token compute scale with how much you have read. A fixed-size state removes both terms: memory stays constant no matter how long the input gets, and each update costs the same at token 1,000,000 as at token 100.

The mechanism, described in the [Kimi Linear paper](https://arxiv.org/abs/2510.26692), extends a line of work called DeltaNet. The delta rule writes to memory the way you would correct a ledger: before storing a new value for a key, it subtracts out whatever the state previously associated with that key, so memory gets overwritten rather than endlessly summed. Gated DeltaNet added a forgetting gate on top, one decay rate per attention head. KDA’s contribution is making that gate channel-wise: every feature dimension in the state gets its own independent forgetting rate, which lets the model hold some features for the whole sequence while aggressively decaying others.

The three designs sit at different points on a memory-precision trade:

| Design | Memory at 1M tokens | Per-token decode cost | Recall behavior |
|---|---|---|---|
| Full attention | KV cache grows with every token | Grows with input length | Exact lookup over all tokens |
| Linear attention (KDA) | Fixed-size state | Constant | Lossy; state must forget to store |
| Hybrid (K3, Kimi Linear) | Cache from full layers only, up to 75% smaller | Mostly constant | Full layers carry precise lookup |

The phrase “must forget to store” is not rhetorical. A fixed-size matrix holding a summary of 800,000 tokens has no way to add the 800,001st without displacing something. This is [context compression](/glossary/context-compression) implemented in the architecture itself, running at every KDA layer, with retention decisions made by learned gates the developer never sees.

KDA models do not abandon full attention; they ration it. Kimi Linear, the 48B-parameter architecture Moonshot published as the proof of concept, interleaves three KDA layers with one full attention layer (multi-head latent attention), which is where the reported numbers come from: KV cache reduced by up to 75%, because only a quarter of the layers keep one, and decode throughput up to 6x at 1M tokens. K3 pairs KDA with gated MLA layers and adds AttnRes on top. Moonshot has not published K3’s exact layer ratio, but KDA is the named backbone, and the 6.3x decoding figure only falls out of a design where full attention is the exception rather than the rule.

To be fair to the design, Moonshot’s evaluations show the hybrid matching or beating full attention under fair comparisons, including on long-context benchmarks, and K3’s BrowseComp and MCP Atlas results suggest it holds up on exactly the multi-step agentic work where context accumulates fastest. Linear attention in 2026 is not the accuracy compromise it was in 2023.

But the design is also an admission worth reading plainly: attention over a million tokens is too expensive to actually pay for, so the strongest open model ever released pays it on a fraction of its layers and approximates it everywhere else. When a detail from hour one of an agent session needs to be recalled at hour six, either a sparse set of full-attention layers finds it in their cache, or it has survived hundreds of thousands of gated updates inside a fixed-size state. Aggregate benchmarks average over both paths. A specific fact your agent needs has to get lucky on one of them.

Nothing about faster decoding fixes what the long-context data already shows. The failure evidence predates K3 and is model-agnostic: a [172-billion-token study across 35 open models](/blog/long-context-tripled-hallucinations-172b-token-study/) found fabrication roughly tripling between 32K and 128K tokens, with every tested model exceeding 10% fabrication at 200K. Chroma’s [context rot](/glossary/context-rot) evaluations found the same shape on frontier closed models, and the [long-context versus RAG data](/blog/long-context-vs-rag-what-the-data-shows/) has pointed the same direction for two years. Those studies measured full-attention models. Nothing in a hybrid design makes the curve bend the other way.

K3’s own release numbers carry the same caution. On AA-Omniscience, K3’s accuracy jumped from 33% to 46% over K2.6 while its hallucination rate climbed from 39% to 51%: the model answers more and abstains less, which shifts more of the reliability burden onto whatever grounding is in its context.

The risk in K3’s pricing is behavioral. At $0.30 per million cached input tokens, [prompt caching](/glossary/prompt-caching) makes re-sending an enormous static context nearly free, and a 1M window invites exactly that: load the whole corpus once, cache it, stop thinking about selection. Cost has quietly been the forcing function behind most context discipline; teams trimmed context because tokens were expensive. When the price of a full window drops by 10x, the discipline has to come from the accuracy data instead, and the accuracy data says the trimming was never really about cost.

Treat K3’s window as a budget, not a warehouse. The advertised 1M is a ceiling, and every empirical result on long context says realized accuracy peaks far below it. That advice predates this model. What K3 adds is a sharper version of the argument: with fixed-state layers doing most of the sequence mixing, whatever you stream into the window gets compressed by the model’s own learned gates, and you do not control what those gates discard. Curating what enters the window is the only selection step the developer actually owns. Skip it, and selection still happens; it just happens inside a recurrent state, silently, at write time.

Concretely, the pattern that wins is [context offloading](/glossary/context-offloading): context that has to be right lives outside the window, in a store that keeps it lossless, and the window holds only the slice the current step needs. The distinction matters more on this architecture than on any before it, because inside K3’s window the alternative to offloading is not “the model remembers everything,” it is compression by gates you cannot inspect. An agent reading from a Wire [container](/glossary/context-container) pulls 5K to 10K tokens of matched context per search from a store that can hold millions, so effective context stays in the range where every architecture’s recall is still precise. Treat the window [like a cache over a larger store](/blog/context-window-demand-paging/), paging context in when the task needs it and evicting it when the task moves on. And keep the model’s job at read time small: five relevant chunks beat a cached half-million-token dump on every accuracy benchmark that has measured the comparison.

Kimi Delta Attention is genuinely good news for that pattern, just not for the reason the launch coverage implies. Cheaper attention does not make context stuffing safe; it makes well-curated context cheaper to serve. A 6.3x faster decode on 20K tokens of selected context is the compounding win. The lesson from every long-context study since 2023 survives its third architecture generation intact: send less to the window, whatever the window costs.

Sources: [Kimi Linear: An Expressive, Efficient Attention Architecture (arXiv 2510.26692)](https://arxiv.org/abs/2510.26692) · [Moonshot AI Releases Kimi K3 (MarkTechPost)](https://www.marktechpost.com/2026/07/16/moonshot-ai-releases-kimi-k3-a-2-8-trillion-parameter-open-moe-model-with-kimi-delta-attention-and-1m-context/) · [Kimi K3, and what we can still learn from the pelican benchmark (Simon Willison)](https://simonwillison.net/2026/Jul/16/kimi-k3/) · [Kimi’s open model K3 nears GPT-5.6 Sol and Fable 5 (The Decoder)](https://the-decoder.com/kimis-open-model-k3-nears-gpt-5-6-sol-and-fable-5-while-signaling-the-end-of-super-cheap-chinese-ai/)

Wire transforms your documents into structured, AI-optimized context containers. Upload files, get MCP tools instantly.