What Comes After LLMs? Mamba, Diffusion & World Models

A developer's analysis argues that transformer-based large language models are approaching fundamental limits, and that hybrid post-transformer systems are already emerging in research labs. The article highlights four key limitations of transformers—quadratic complexity, data exhaustion, compute costs, and lack of reasoning—and points to alternative architectures like state space models (Mamba) and world models (JEPA) as candidates for the next shift. The author notes that Yann LeCun has been advocating for world models over word predictors, and that the evidence for this transition is concrete.

AI hype follows a rhythm: roughly every three years, the architecture everyone is betting on gets replaced by something newer. The next shift is already forming in research labs. What follows is a tour of the strongest candidates to replace the transformer, and of the evidence that the shift is already under way. The pattern repeats: a dominant architecture, a wall it cannot scale past, and a new design that routes around that wall. This article walks through the four limitations closing in on transformers, the three alternative architectures, and what the hybrid systems already in production tell us. The goal is a practical read for engineers deciding what to architect for 2027 and beyond — not a prediction market, just the evidence as it stands in 2026. Every few years the center of gravity of AI shifts: php flowchart LR A "~2017<br/ Transformers replace<br/ RNNs and CNNs" -- B "2020-2022<br/ Diffusion models take<br/ image generation" B -- C "2022<br/ LLMs reach everyone<br/ GPT-3, ChatGPT " C -- D "2026-2028<br/ Hybrid post-transformer<br/ systems in production" Transformers showed up around 2017 and made RNNs and CNNs look like relics. Diffusion models then took over image generation between 2020 and 2022. And LLMs exploded into everything starting in 2022 with GPT-3 and ChatGPT. By that pattern, we should be approaching the next shift. It is already happening — just not where you would expect. Not on social media or in press releases, but in research labs. Yann LeCun spent 2025 arguing that autoregressive LLMs are fundamentally limited. "We need world models, not word predictors," he said at conference after conference. Meanwhile his team published a sequence of papers on Joint Embedding Predictive Architecture JEPA — first for images, then video, then language in September 2025. The supporting evidence from the last twelve months is unusually concrete: Transformers are incredible at what they do. But four problems are closing in that no amount of scaling can fix. Self-attention means every token talks to every other token — O n² complexity. Double your context and you quadruple the compute. At 10 million tokens, the math simply breaks. You can throw more GPUs at it, but you are fighting basic arithmetic. LLMs train on internet text, and by 2026 most of it has been scraped. Books, Wikipedia, Reddit, GitHub — all consumed. Synthetic data sounds like a fix until you realize that models trained on AI-generated text degrade over generations: they learn the artifacts and biases of the generation process itself. The scaling laws that powered everything from 2020 to 2024 are hitting diminishing returns. Frontier models cost tens or hundreds of millions to train now. Serving billions of requests daily burns massive compute. AI data centers need gigawatts of power, and power generation capacity is becoming a bottleneck. Sam Altman admitted in 2024 that compute costs limit everything — and that has not changed. If the constraint is energy, the answer can come from the model side or the silicon side. My notes on cost optimization for LLM systems https://www.glukhov.org/llm-architecture/cost-optimization/cost-optimization-for-llm-systems/ and on specialized LLM inference chips https://www.glukhov.org/llm-performance/hardware/llm-asics/ cover each side of that tradeoff. LLMs are pattern matchers at scale, predicting the next token from training data. Exceptional at it, sure — but not reasoning, and not understanding causal relationships in the physical world. If you want AI that can plan, simulate, or genuinely adapt to new environments, token prediction will not get you there. State space models SSMs such as Mamba and Gated DeltaNet take a completely different approach to sequences. Instead of attending to all previous tokens, they maintain a compressed internal state that summarizes history — linear O n scaling instead of quadratic, with constant memory per token at inference regardless of how long the sequence gets. The mechanism behind that efficiency is worth understanding, because it explains both the speed and the tradeoff. Classic state space models the S4 family were time-invariant: the same fixed transition matrix applied at every step, which is fast but cannot decide that one token matters more than another. Mamba's contribution was to make those transition parameters selective — functions of the current input — so the model can dynamically choose what to keep in its compressed state and what to discard, much like attention's softmax decides what to emphasize, but without materializing an O n² score matrix. To keep that selective recurrence fast, Mamba runs it as a hardware-aware parallel scan: an associative operation computed in GPU SRAM rather than a token-by-token loop, which is the same kind of low-level kernel engineering that FlashAttention did for transformers. Mamba-2 pushed this further with "state space duality," showing that a selective SSM and a restricted form of linear attention compute the same underlying operation — letting SSM kernels reuse the matrix-multiplication hardware that GPUs are optimized for. The real production breakthrough here was Gated DeltaNet , an improvement on Mamba2 that uses a delta rule for better long-context handling. Qwen3.5 adopted it as its core architecture in February 2026 and got strong results on agent tasks and coding benchmarks using only 3 billion active parameters in its hybrid MoE configuration. That is impressive efficiency, and my llama.cpp benchmarks of Qwen 3.6 MTP versus standard decoding https://www.glukhov.org/llm-performance/benchmarks/comparing-qwen-3-6-mtp-vs-standard/ on a 16 GB GPU give a sense of how Qwen models behave on consumer hardware. The tradeoff is structural, not just a training artifact: because an SSM compresses the entire history into a fixed-size state, it can lose fine-grained detail that transformers preserve by keeping every past token individually addressable through attention. If you need exact copying or citation, transformers still win. For most practical workloads — summarization, agentic tool use, long-document reasoning where the gist matters more than verbatim recall — the efficiency gain is worth it. Diffusion language models apply the same technique that transformed image generation to text. Instead of generating one token at a time, they can produce multiple tokens in parallel during each forward pass. NVIDIA's Nemotron-Labs-Diffusion , released in May 2026, achieved nearly 6x tokens per forward pass compared to Qwen3-8B while maintaining competitive accuracy across benchmarks including HumanEval, GSM8K, and Math500. It is a tri-mode model that unifies autoregressive, diffusion, and self-speculation decoding. Google's Gemini Diffusion promises even more speedup. The catch is quality versus speed. Autoregressive models optimize for local coherence at each step — the most probable next token. Diffusion models optimize globally across the entire output, which can improve overall structure but sometimes sacrifices fine-grained detail. For creative writing or brainstorming, diffusion might be better. For precise technical output, autoregressive still wins. It is worth separating this from inference-time tricks. Speculative decoding https://www.glukhov.org/llm-performance/optimization/speculative-decoding/ already delivers several tokens per forward pass by drafting and verifying, with no change to the output distribution. Diffusion moves parallel token generation from the inference layer into the model architecture itself — a structurally different bet. World models, specifically Yann LeCun's JEPA architecture, take the most radical approach. Instead of predicting the next token, they predict the next latent embedding — a compressed representation of what should come next. That is closer to how human cognition works: we do not predict individual words, we predict meaning. The architectural difference from both transformers and diffusion models is the missing decoder. A generative model — autoregressive or diffusion — has to reconstruct its target in the original space: exact pixels, exact tokens. That forces the model to spend capacity modeling unpredictable surface noise: the exact phrasing of a sentence, the exact pixel value of a leaf in the wind. JEPA's loss operates entirely inside a learned representation space instead, comparing a predicted embedding against a target embedding produced by a separate, slowly-updated encoder. Nothing is ever decoded back to raw output during training. That lets the predictor discard the noise and keep only the structural, task-relevant part of the signal — which is also, not coincidentally, the part that is useful for planning: an agent can simulate "what happens if I do X" by rolling forward in latent space, without paying the cost of rendering a full image or generating full text for every hypothetical. VL-JEPA achieved performance comparable to established vision-language models with only 1.6 billion parameters versus 7 billion or more for competitors. It reduces decoding operations by about 2.85x while improving accuracy on visual question answering. More importantly, JEPA architectures are designed for continual learning — they build internal models of how the environment works, not just statistical patterns in text. The limitation is that JEPA is still maturing for pure language tasks: LLM-JEPA exists but has not matched large autoregressive or diffusion models on standard benchmarks yet, and JEPA is not a drop-in replacement for a chatbot — it is a representation-learning and planning substrate that complements language models more than it competes with them head-on. For embodied AI and robotics, however — where understanding physical causality matters more than generating fluent prose — JEPA might be the right architecture from the start. The most interesting developments combine elements from multiple architectures. AI21 Labs' Jamba interleaves transformer attention layers with Mamba state space layers — attention for long-range dependencies, SSMs for efficient local processing. Qwen3.5 combines Gated DeltaNet with Mixture of Experts, activating only the relevant expert networks for each input. Cloud providers are already listing Mamba-based models — Mistral's Codestral Mamba was the first open-source Mamba-2 release — and the cloud LLM providers overview https://www.glukhov.org/llm-hosting/cloud/cloud-llm-providers/ tracks what is available where. Different tasks have different requirements: A single architecture will not solve everything. The same logic that drives multi-model system design https://www.glukhov.org/llm-architecture/model-routing/multi-model-system-design/ — the right mechanism for the right task — is now appearing inside the models themselves. If you are building AI applications and architecting for 2027 or 2028, do not over-invest in pure transformer solutions. Consider whether SSM or diffusion components might better serve your use case. The ecosystem is shifting, and early adopters will have an advantage. Two further implications are worth tracking. Qwen3.5's performance with only 3 billion active parameters suggests that specialized, efficient architectures will outperform larger general-purpose models for many tasks — which could democratize AI deployment and make high-performance models accessible on consumer hardware. And diffusion language models could reduce latency by an order of magnitude for certain workloads; applications requiring real-time response — conversational agents, live translation, interactive tools — will benefit most. Following the three-year cycle, I would expect the first production deployments of non-transformer architectures by 2027–2028 . These will likely be hybrid systems that leverage transformer attention where needed but use SSMs or diffusion for efficiency gains. The consolidation phase we are in now — where labs prove that LLMs can reason and act — is giving way to a harder challenge: making these systems efficient, adaptable, and spatially aware. That demands breakthroughs in continual learning, world models, and architectural efficiency. Each addresses fundamental limitations that pure scaling cannot solve. The next big thing will not be bigger models. It will be smarter architectures that achieve more with less — models that understand rather than just predict, that adapt rather than just process. That is what comes after LLMs. Not a single new architecture, but a generation of specialized, efficient systems that finally move beyond pattern matching toward genuine understanding.