What Comes After LLMs? Mamba, Diffusion & World Models

AI hype follows a rhythm: roughly every three years, the architecture everyone is betting on gets replaced by something newer. The next shift is already forming in research labs.

What follows is a tour of the strongest candidates to replace the transformer, and of the evidence that the shift is already under way. The pattern repeats: a dominant architecture, a wall it cannot scale past, and a new design that routes around that wall.

This article walks through the four limitations closing in on transformers, the three alternative architectures, and what the hybrid systems already in production tell us. The goal is a practical read for engineers deciding what to architect for 2027 and beyond — not a prediction market, just the evidence as it stands in 2026.

The pattern: AI’s three-year breakthrough cycle

Every few years the center of gravity of AI shifts:

Transformers showed up around 2017 and made RNNs and CNNs look like relics. Diffusion models then took over image generation between 2020 and 2022. And LLMs exploded into everything starting in 2022 with GPT-3 and ChatGPT.

By that pattern, we should be approaching the next shift. It is already happening — just not where you would expect. Not on social media or in press releases, but in research labs.

Yann LeCun spent 2025 arguing that autoregressive LLMs are fundamentally limited. “We need world models, not word predictors,” he said at conference after conference. Meanwhile his team published a sequence of papers on Joint Embedding Predictive Architecture (JEPA) — first for images, then video, then language in September 2025.

The supporting evidence from the last twelve months is unusually concrete:

• State space models like Mamba handle million-token contexts with linear scaling — something transformers cannot do because of their quadratic attention mechanism.
• Google announced Gemini Diffusion, promising text generation up to 10x faster.
• NVIDIA shipped Nemotron 3 with Mamba-2 layers built in.
• China’s Qwen3.5 went live using Gated DeltaNets.
• In December 2025, VL-JEPA landed — a vision-language model with half the parameters of its competitors that still performed better.

Why transformers are running out of road

Transformers are incredible at what they do. But four problems are closing in that no amount of scaling can fix.

Quadratic attention breaks at long context

Self-attention means every token talks to every other token — O(n²) complexity. Double your context and you quadruple the compute. At 10 million tokens, the math simply breaks. You can throw more GPUs at it, but you are fighting basic arithmetic.

The training data is running out

LLMs train on internet text, and by 2026 most of it has been scraped. Books, Wikipedia, Reddit, GitHub — all consumed. Synthetic data sounds like a fix until you realize that models trained on AI-generated text degrade over generations: they learn the artifacts and biases of the generation process itself. The scaling laws that powered everything from 2020 to 2024 are hitting diminishing returns.

Energy and cost are the hard ceiling

Frontier models cost tens or hundreds of millions to train now. Serving billions of requests daily burns massive compute. AI data centers need gigawatts of power, and power generation capacity is becoming a bottleneck. Sam Altman admitted in 2024 that compute costs limit everything — and that has not changed.

If the constraint is energy, the answer can come from the model side or the silicon side. My notes on cost optimization for LLM systems and on specialized LLM inference chips cover each side of that tradeoff.

Pattern matching is not understanding

LLMs are pattern matchers at scale, predicting the next token from training data. Exceptional at it, sure — but not reasoning, and not understanding causal relationships in the physical world. If you want AI that can plan, simulate, or genuinely adapt to new environments, token prediction will not get you there.

State space models: linear scaling with a compressed memory

State space models (SSMs) such as Mamba and Gated DeltaNet take a completely different approach to sequences. Instead of attending to all previous tokens, they maintain a compressed internal state that summarizes history — linear O(n) scaling instead of quadratic, with constant memory per token at inference regardless of how long the sequence gets.

The mechanism behind that efficiency is worth understanding, because it explains both the speed and the tradeoff. Classic state space models (the S4 family) were time-invariant: the same fixed transition matrix applied at every step, which is fast but cannot decide that one token matters more than another. Mamba’s contribution was to make those transition parameters selective — functions of the current input — so the model can dynamically choose what to keep in its compressed state and what to discard, much like attention’s softmax decides what to emphasize, but without materializing an O(n²) score matrix. To keep that selective recurrence fast, Mamba runs it as a hardware-aware parallel scan: an associative operation computed in GPU SRAM rather than a token-by-token loop, which is the same kind of low-level kernel engineering that FlashAttention did for transformers. Mamba-2 pushed this further with “state space duality,” showing that a selective SSM and a restricted form of linear attention compute the same underlying operation — letting SSM kernels reuse the matrix-multiplication hardware that GPUs are optimized for.

The real production breakthrough here was Gated DeltaNet, an improvement on Mamba2 that uses a delta rule for better long-context handling. Qwen3.5 adopted it as its core architecture in February 2026 and got strong results on agent tasks and coding benchmarks using only 3 billion active parameters in its hybrid MoE configuration. That is impressive efficiency, and my llama.cpp benchmarks of Qwen 3.6 MTP versus standard decoding on a 16 GB GPU give a sense of how Qwen models behave on consumer hardware.

The tradeoff is structural, not just a training artifact: because an SSM compresses the entire history into a fixed-size state, it can lose fine-grained detail that transformers preserve by keeping every past token individually addressable through attention. If you need exact copying or citation, transformers still win. For most practical workloads — summarization, agentic tool use, long-document reasoning where the gist matters more than verbatim recall — the efficiency gain is worth it.

Diffusion language models: many tokens per forward pass

Diffusion language models apply the same technique that transformed image generation to text. Instead of generating one token at a time, they can produce multiple tokens in parallel during each forward pass.

NVIDIA’s Nemotron-Labs-Diffusion, released in May 2026, achieved nearly 6x tokens per forward pass compared to Qwen3-8B while maintaining competitive accuracy across benchmarks including HumanEval, GSM8K, and Math500. It is a tri-mode model that unifies autoregressive, diffusion, and self-speculation decoding. Google’s Gemini Diffusion promises even more speedup.

The catch is quality versus speed. Autoregressive models optimize for local coherence at each step — the most probable next token. Diffusion models optimize globally across the entire output, which can improve overall structure but sometimes sacrifices fine-grained detail. For creative writing or brainstorming, diffusion might be better. For precise technical output, autoregressive still wins.

It is worth separating this from inference-time tricks. Speculative decoding already delivers several tokens per forward pass by drafting and verifying, with no change to the output distribution. Diffusion moves parallel token generation from the inference layer into the model architecture itself — a structurally different bet.

World models and JEPA: predicting meaning, not tokens

World models, specifically Yann LeCun’s JEPA architecture, take the most radical approach. Instead of predicting the next token, they predict the next latent embedding — a compressed representation of what should come next. That is closer to how human cognition works: we do not predict individual words, we predict meaning.

The architectural difference from both transformers and diffusion models is the missing decoder. A generative model — autoregressive or diffusion — has to reconstruct its target in the original space: exact pixels, exact tokens. That forces the model to spend capacity modeling unpredictable surface noise: the exact phrasing of a sentence, the exact pixel value of a leaf in the wind. JEPA’s loss operates entirely inside a learned representation space instead, comparing a predicted embedding against a target embedding produced by a separate, slowly-updated encoder. Nothing is ever decoded back to raw output during training. That lets the predictor discard the noise and keep only the structural, task-relevant part of the signal — which is also, not coincidentally, the part that is useful for planning: an agent can simulate “what happens if I do X” by rolling forward in latent space, without paying the cost of rendering a full image or generating full text for every hypothetical.

VL-JEPA achieved performance comparable to established vision-language models with only 1.6 billion parameters versus 7 billion or more for competitors. It reduces decoding operations by about 2.85x while improving accuracy on visual question answering. More importantly, JEPA architectures are designed for continual learning — they build internal models of how the environment works, not just statistical patterns in text.

The limitation is that JEPA is still maturing for pure language tasks: LLM-JEPA exists but has not matched large autoregressive or diffusion models on standard benchmarks yet, and JEPA is not a drop-in replacement for a chatbot — it is a representation-learning and planning substrate that complements language models more than it competes with them head-on. For embodied AI and robotics, however — where understanding physical causality matters more than generating fluent prose — JEPA might be the right architecture from the start.

What production systems are doing: hybrid architectures

The most interesting developments combine elements from multiple architectures. AI21 Labs’ Jamba interleaves transformer attention layers with Mamba state space layers — attention for long-range dependencies, SSMs for efficient local processing. Qwen3.5 combines Gated DeltaNet with Mixture of Experts, activating only the relevant expert networks for each input. Cloud providers are already listing Mamba-based models — Mistral’s Codestral Mamba was the first open-source Mamba-2 release — and the cloud LLM providers overview tracks what is available where.

Different tasks have different requirements:

• Code generation might benefit from diffusion’s parallel token generation.
• Long-document analysis needs an SSM’s linear scaling.
• Creative writing could use transformer attention for coherence.

A single architecture will not solve everything. The same logic that drives multi-model system design — the right mechanism for the right task — is now appearing inside the models themselves.

If you are building AI applications and architecting for 2027 or 2028, do not over-invest in pure transformer solutions. Consider whether SSM or diffusion components might better serve your use case. The ecosystem is shifting, and early adopters will have an advantage.

Two further implications are worth tracking. Qwen3.5’s performance with only 3 billion active parameters suggests that specialized, efficient architectures will outperform larger general-purpose models for many tasks — which could democratize AI deployment and make high-performance models accessible on consumer hardware. And diffusion language models could reduce latency by an order of magnitude for certain workloads; applications requiring real-time response — conversational agents, live translation, interactive tools — will benefit most.

Where this is heading: the post-transformer era

Following the three-year cycle, I would expect the first production deployments of non-transformer architectures by 2027–2028. These will likely be hybrid systems that leverage transformer attention where needed but use SSMs or diffusion for efficiency gains.

The consolidation phase we are in now — where labs prove that LLMs can reason and act — is giving way to a harder challenge: making these systems efficient, adaptable, and spatially aware. That demands breakthroughs in continual learning, world models, and architectural efficiency. Each addresses fundamental limitations that pure scaling cannot solve.

The next big thing will not be bigger models. It will be smarter architectures that achieve more with less — models that understand rather than just predict, that adapt rather than just process. That is what comes after LLMs. Not a single new architecture, but a generation of specialized, efficient systems that finally move beyond pattern matching toward genuine understanding.

References

1. Adaline Labs. “The AI Research Landscape in 2026: From Agentic AI to Embodiment.” https://labs.adaline.ai/p/the-ai-research-landscape-in-2026
2. Aftab, A. “The End of LLMs As We Know Them: Why 2026 Marks the Beginning of AI’s Next Architecture Revolution.” Medium, 2025. https://medium.com/@aftab001x/the-end-of-llms-as-we-know-them-why-2026-marks-the-beginning-of-ais-next-architecture-revolution-902ee29484f7
3. Greengard, S. “Beyond LLMs: A Post-Transformer World Emerges.” Communications of the ACM, May 2026. https://cacm.acm.org/news/beyond-llms-a-post-transformer-world-emerges
4. NVIDIA. “Nemotron-Labs-Diffusion: A Tri-Mode Language Model Unifying Autoregressive, Diffusion, and Self-Speculation Decoding.” arXiv:2607.05722, 2026. https://arxiv.org/html/2607.05722v1
5. MarkTechPost. “NVIDIA AI Releases Nemotron-Labs-Diffusion: A Tri-Mode Language Model with 6× Tokens Per Forward Over Qwen3-8B.” May 20, 2026. https://www.marktechpost.com/2026/05/20/nvidia-ai-releases-nemotron-labs-diffusion-a-tri-mode-language-model-with-6x-tokens-per-forward-over-qwen3-8b
6. Chen, D. et al. “VL-JEPA: Joint Embedding Predictive Architecture for Vision-language.” arXiv:2512.10942, 2025. https://arxiv.org/pdf/2512.10942
7. OpenReview. “VL-JEPA: Joint Embedding Predictive Architecture for Vision-language.” https://openreview.net/forum?id=tjimrqc2BU
8. NVIDIA Research. “Gated Delta Networks: Improving Mamba2 with Delta Rule.” ICLR 2025. https://research.nvidia.com/publication/2025-04_gated-delta-networks-improving-mamba2-delta-rule
9. Laon People. “Why Did Qwen3.5 Choose Gated DeltaNet?” February 2026. https://laonpeople.com/en/blog/why-did-qwen3-5-choose-gated-deltanet
10. Google DeepMind. “Gemini Diffusion.” https://deepmind.google/models/gemini-diffusion
11. Vicentino, C. “Autoregressive vs. Masked Diffusion Language Models: A Controlled Comparison.” arXiv:2603.22075, March 2026. https://www.alphaxiv.org/abs/2603.22075
12. JetBrains AI Blog. “Why Diffusion Models Could Change Developer Workflows in 2026.” November 2025. https://blog.jetbrains.com/ai/2025/11/why-diffusion-models-could-change-developer-workflows-in-2026
13. Spheron Blog. “Mamba-3 and State Space Models on GPU Cloud: Deploy SSM Inference as the Transformer Alternative (2026 Guide).” https://www.spheron.network/blog/mamba-3-state-space-model-gpu-cloud-deployment
14. Gartner. “Gartner Predicts 40 Percent of Enterprise Apps Will Feature Task-Specific AI Agents by 2026.” August 26, 2025. https://www.gartner.com/en/newsroom/press-releases/2025-08-26-gartner-predicts-40-percent-of-enterprise-apps-will-feature-task-specific-ai-agents-by-2026-up-from-less-than-5-percent-in-2025
15. Reddit r/deeplearning. “Following a 3-year AI breakthrough cycle.” https://www.reddit.com/r/deeplearning/comments/1k8gdwg/following_a_3year_ai_breakthrough_cycle
16. Gu, A., Dao, T. “Mamba: Linear-Time Sequence Modeling with Selective State Spaces.” arXiv:2312.00752, 2023. https://doi.org/10.48550/arxiv.2312.00752
17. Dao, T., Gu, A. “Transformers are SSMs: Generalized Models and Efficient Algorithms Through Structured State Space Duality (Mamba-2).” arXiv:2405.21060, 2024. https://arxiv.org/abs/2405.21060
18. Meta AI Blog. “The first AI model based on Yann LeCun’s vision for more human-like AI (I-JEPA).” https://ai.meta.com/blog/yann-lecun-ai-model-i-jepa/
19. LeCun, Y. “A Path Towards Autonomous Machine Intelligence.” Position paper, 2022. https://openreview.net/forum?id=BZ5a1r-kVsf