{"slug": "from-full-to-hybrid-attention-in-llms", "title": "From Full to Hybrid Attention in LLMs", "summary": "In the second half of 2026, half of new open LLMs use no full attention at all, and the remaining half mix full global attention with sliding-window, Gated DeltaNet, KDA, Mamba-2, Lightning Attention, convolution or sparse layers, according to an analysis by Sebastian Raschka. The shift away from full attention throughout — global MHA, GQA, dense MLA or CCA in every layer — follows 2025's move to \"some full attention\" hybrids and continued through the first half of 2026, and Raschka attributes it to making long contexts cheaper: sparse attention cuts attention computation by selecting token or block subsets, while recurrent layers such as Gated DeltaNet hold a fixed-size state to save memory at the cost of exact retrieval.", "body_md": "# From Full to Hybrid Attention in LLMs\n\nCurrently working towards a bigger open LLM roundup. But in the meantime, just one of the interesting trends that stood out. I.e., looks like the transition from full to hybrid attention continues.\n\nIn 2025, there were still lots of full attention models, with a shift to “some full attention” hybrids.\n\nIn the first half of 2026, the trend towards “some full attention” hybrids continued.\n\nAnd now, in the second half of 2026, we have a 50/50 split between “some full attention” hybrids and models that don’t use full attention at all anymore. And yes, there are also no full attention models anymore.\n\nThe plot legend here means,\n\n- \nFull attention throughout = Global MHA, GQA, dense MLA, CCA in EVERY layer.\n- \nSome full attention = Full global attn mixed with: sliding-window/chunked attention, Gated DeltaNet, KDA, Mamba-2, Lightning Attention, convolutions, sparse attention.\n- \nNo full attention = Zero dense global layers.\n\nThe main implication is that the trend goes towards making long contexts cheaper to use. This means, it makes the architectures more suitable for long documents and conversations, large code bases, and of course agent use cases that rely on a lot of tool calls.\n\nIn particular, there are two distinct efficiency gains here:\n\n1. \nLess attention computation, since sparse attention and the like select tokens or blocks from the context and perform the expensive attention operation from only that subset. (Note that it can still retrieve information from distant positions.)\n2. \nLess memory for retaining context in some architectures, meaning that recurrent layers such as Gated DeltaNet maintain a fixed-size state. Of course, this saves memory, but compressing the history creates challenges for exact retrieval, so they are usually intermixed with traditional attention methods (sparse or full).\n\nSource: lightly edited website version of my [Substack note](https://substack.com/@rasbt/note/c-383850419).\n\n## Read Next\n\n[Open October LLM Architecture Roundup A roundup of ten open-weight LLM architecture releases from October 2026, covering attention mechanisms, hybrid layers, and MoE designs.](https://sebastianraschka.com/blog/2026/open-october-llm-architecture-releases.html)\n\n[Focusing on Post-Training Why I'd invest in post-training existing open-weight LLMs, with Fireworks' Ember-1 as an example of more token-efficient reasoning.](https://sebastianraschka.com/blog/2026/focusing-on-llm-post-training.html)\n\n[MiMo-V2.6 Pro Architecture and Training Notes Notes on MiMo-V2.6 Pro's GQA and sliding-window attention, agent training tasks, reward signals, and large RL batches.](https://sebastianraschka.com/blog/2026/mimo-v2-6-pro-architecture-training-notes.html)", "url": "https://wpnews.pro/news/from-full-to-hybrid-attention-in-llms", "canonical_source": "https://sebastianraschka.com/blog/2026/full-to-hybrid-attention.html", "published_at": "2026-11-10 16:31:49+00:00", "updated_at": "2026-11-10 16:56:40.698294+00:00", "lang": "en", "topics": ["large-language-models", "artificial-intelligence", "machine-learning", "ai-research"], "entities": ["Sebastian Raschka", "Gated DeltaNet", "Mamba-2", "Lightning Attention", "KDA", "Substack"], "also_reported_by": [], "alternates": {"html": "https://wpnews.pro/news/from-full-to-hybrid-attention-in-llms", "markdown": "https://wpnews.pro/news/from-full-to-hybrid-attention-in-llms.md", "text": "https://wpnews.pro/news/from-full-to-hybrid-attention-in-llms.txt", "jsonld": "https://wpnews.pro/news/from-full-to-hybrid-attention-in-llms.jsonld"}}