The Attention Rebuild: How 2026's Open Models Made 1M Context Fit on Machines

In 2026, open-weight models such as Kimi K3, DeepSeek V4, Inkling, and Nemotron 3 advertise million-token context windows, made feasible on consumer hardware by rebuilding attention mechanisms. A 48GB Mac can now run Qwen 3.6 27B at its full 262k context, whereas a dense Llama 3.3 70B would require about 655GB of KV cache for a million tokens. Techniques include sliding window attention, linear attention (e.g., Kimi Delta Attention cutting KV cache by up to 75%), and hybrid architectures with periodic full-attention layers.

Every headline model of 2026 advertises a million-token context window: Kimi K3, DeepSeek V4, Inkling, Nemotron 3. Not long ago that number would have been a joke for local hardware, and our own KV cache explainer https://vettedconsumer.com/the-kv-cache-explained-why-long-context-eats-your-vram-and-how-to-fit-more/ spelled out why: on a classic dense transformer, context is a memory tax that grows with every token. Run the math on a Llama 3.3 70B and it is brutal. Its cache costs about 0.66MB per token 80 layers, 8 KV heads, 128-dim heads, 16-bit , so 128k tokens of context is roughly 84GB , and a million tokens would be around 655GB, more memory than the model itself . Yet here is a measured, real-world 2026 result: a 48GB Mac runs Qwen 3.6 27B at its full 262k context , model and cache together, with room left for macOS. Nothing about that machine changed. The models changed. While everyone was watching the MoE takeover https://vettedconsumer.com/every-frontier-open-model-is-a-moe-now-what-that-does-to-your-hardware-math/ , the same labs quietly made a second, matching bet: they rebuilt attention itself. This piece is about that second convergence, because it decides how much context your machine can really hold. The problem: full attention pays rent on every token In a standard transformer, every layer stores a key and value vector for every token you have ever fed it, and every new token looks back at all of them. That store is the KV cache. It grows linearly with context length, multiplied across every layer, and it lives in your fast memory alongside the weights. Double your context, double the tax. This design has perfect recall, and a price that made long context a datacenter luxury. The 2026 releases attack that price from three directions, usually in combination: 1. Sliding windows: only remember what's nearby. Most layers watch only the last few thousand tokens, so their cache stops growing at the window size. Mistral 7B https://arxiv.org/abs/2310.06825?ref=vettedconsumer.com Jiang et al., 2023 mainstreamed this in open models, using sliding window attention "to effectively handle sequences of arbitrary length with a reduced inference cost." 2. Linear attention: replace the cache with a fixed-size state. Instead of storing everything, a recurrent-style layer compresses history into a state whose size never grows, the lineage of Mamba https://arxiv.org/abs/2312.00752?ref=vettedconsumer.com Gu & Dao, 2023 , which showed "linear scaling in sequence length" with a constant-size state and roughly 5x inference throughput over comparable transformers. The modern refinement is Moonshot's Kimi Delta Attention https://arxiv.org/abs/2510.26692?ref=vettedconsumer.com the Kimi Linear paper , a gated linear-attention module that cuts KV cache usage "by up to 75%" in long-sequence generation. 3. Keep a few full-attention layers as the safety net. Pure linear attention historically loses precise long-range recall, so the winning recipe is a hybrid: mostly cheap layers, with periodic full-attention layers preserving global lookback. Kimi Linear interleaves them 3:1; Inkling runs 55 sliding-window layers against 11 global ones. Who is running what | 2026 model | Attention design | What it does to context cost | |---|---|---| | Inkling https://vettedconsumer.com/inkling-mira-muratis-first-open-model-is-a-975b-moe-you-can-actually-run/ 975B MiniMax M3 https://vettedconsumer.com/minimax-m3-the-first-open-weight-multimodal-frontier-model-and-the-license-catch/ 428B Sources linked in each row and below; "owner-measured" and "vendor-reported" labels matter, and we keep them. Set that table against the dense baseline from the intro and the shift is stark. The old question was "how much context can you afford?" The new question is "does your runtime understand the model's layout?", because on these architectures the cache stopped being the thing that runs out first. What owners measure The best public numbers come from the community member who converted Qwen 3.6 27B for llama.cpp, froggeric on r/LocalLLaMA https://www.reddit.com/r/LocalLLaMA/comments/1t57xuu/25x faster inference with qwen 36 27b using mtp/?ref=vettedconsumer.com , whose fit tables we referenced above: "only 16 of 65 layers use KV cache verified . The other 48 are linear attention fixed 898 MiB recurrent state . KV memory is ~4x less than a standard dense model." His follow-on warning is the practical gotcha of this whole transition: "Runtimes that don't handle this e.g. vllm allocate KV for all 65 layers and show much higher memory usage." The architecture only saves you memory if your runtime knows about it. Support for each hybrid design lands in llama.cpp, vLLM, and MLX at different speeds, which is exactly why the Inkling and K3 GGUFs are still waiting on merged support as we covered in the K3 hardware piece https://vettedconsumer.com/what-hardware-runs-kimi-k3-the-2-8t-options-ranked-and-when-to-just-rent/ , and why a model can look "supported" while quietly allocating dense-sized caches. The fine print: what 1M context still costs Prefill did not get free. The memory tax fell, but reading a million tokens still takes real compute. As we covered in prompt processing vs generation https://vettedconsumer.com/prompt-processing-vs-generation-why-your-box-is-fast-at-one-and-slow-at-the-other/ , prefill is the compute-bound half, and on a prosumer box a truly full window is a wait measured in many minutes, sometimes worse. MiniMax's 14.2x prefill speedup claim exists precisely because this is the new bottleneck. Advertised context and patience-limited context are different numbers. Recall is a spectrum now. A dense-attention model provably sees every token. A hybrid sees recent tokens sharply and older ones through a compressed state or periodic global layers. The Kimi Linear paper argues its hybrid matches or beats full attention on their evaluations, and vendors publish strong needle-in-a-haystack results, but independent long-context evaluation of the 2026 hybrids is still thin. If your workload depends on exact recall of something 800k tokens back, test it before you trust it. Quantizing the cache still helps. On hybrid models the KV that remains can still be squeezed q8 0 cache halves it again, the standard trick from our KV guide https://vettedconsumer.com/the-kv-cache-explained-why-long-context-eats-your-vram-and-how-to-fit-more/ , and owner tables show exactly that combination reaching the full 262k on mid-tier boxes. The decision cheat-sheet | Your situation | What the attention rebuild means for you | |---|---| | Long documents / codebases on a 24 to 48GB box | Pick a hybrid-attention model Qwen 3.6 class : six-figure context genuinely fits where a dense model managed ~32k | | Agent sessions that grow all day | Hybrids keep memory flat as the transcript grows; watch prefill time, not VRAM | | Exact recall over huge distances | Prefer models with more full-attention layers, and test your own needle case; hybrid recall is good but not guaranteed | | Running brand-new hybrids | Check your runtime supports the architecture natively, or your "efficient" model allocates dense-sized cache | | Dense favorites Llama 3.3 70B class | The old math still applies: budget ~0.66MB per token of context and quantize the cache | Put the two 2026 convergences together and the design brief behind this model generation is complete: MoE froze the per-token compute cost while capacity exploded, and the attention rebuild froze the per-token memory cost while context exploded. Both bets favor the same hardware: as much memory capacity as you can buy, fed by decent bandwidth. Whatever box you own, the practical question is the same as ever, and our calculator https://vettedconsumer.com/can-i-run-it/ now answers it with context length as a first-class input. Sources and how we researched this - Research spine: Gu & Dao 2023, "Mamba: Linear-Time Sequence Modeling with Selective State Spaces" arXiv:2312.00752 https://arxiv.org/abs/2312.00752?ref=vettedconsumer.com ; Jiang et al. 2023, "Mistral 7B" arXiv:2310.06825 https://arxiv.org/abs/2310.06825?ref=vettedconsumer.com for sliding window attention in open models; Moonshot AI 2025, "Kimi Linear: An Expressive, Efficient Attention Architecture" arXiv:2510.26692 https://arxiv.org/abs/2510.26692?ref=vettedconsumer.com for Kimi Delta Attention and the 75% KV-reduction figure. - Model architecture details: vendor model cards and announcements as covered in our linked per-model spotlights; Inkling's layer layout per the llama.cpp support PR; MiniMax MSA figures are MiniMax's reported numbers; DeepSeek V4's attention is the vendor's description. - Owner measurements: froggeric's Qwen 3.6 conversion thread https://www.reddit.com/r/LocalLLaMA/comments/1t57xuu/25x faster inference with qwen 36 27b using mtp/?ref=vettedconsumer.com verified KV-layer count, fixed recurrent state, per-tier context tables , quoted verbatim. - The dense-baseline math is the standard KV formula 2 x layers x KV heads x head dim x bytes per value applied to Llama 3.3 70B's published configuration. - We have not benchmarked these architectures first-hand. This synthesizes the papers, vendor documentation, and attributed owner reports, cited inline. Related: The KV cache, explained · Every frontier open model is a MoE now · Prompt processing vs generation · Bandwidth, Not TFLOPS · Speculative decoding, explained