If you have watched an AI write, you know the ritual. Tokens appear left to right, one after another, like someone typing very fast. It feels like proof of intelligence. It is actually a constraint. Every mainstream language model, from GPT to Claude to the small model running on your laptop, is locked into a strictly sequential process: emit a token, condition on it, emit the next one. Never look ahead. Never go back.
That constraint is now being attacked from an unexpected direction. This week, two deep explanatory posts are circulating on Hacker News at the same time: a guide from the Kuleshov group at Cornell titled "How to Build a Diffusion Language Model," and Sander Dieleman's post on continuous diffusion language models. They land on top of a real product wave. Inception Labs' Mercury generates over 1,000 tokens per second per user on standard GPUs. NVIDIA's open-weight Nemotron Diffusion models report 2 to 8 times the throughput of comparable autoregressive models while retaining up to 99 percent of their quality. Google shipped Gemma Diffusion as an open-weights release.
One disclosure before we go further. I am a backend engineer who runs his own AI agent infrastructure, not an ML researcher. I have never trained a diffusion model. Everything below comes from reading the primary sources this week, and I will link them so you can check me. But I found that the core idea is surprisingly buildable once you see it, and the "which one should I care about" question has a concrete answer now. That is what this article is for.
An autoregressive model generates text the way a very strict typist would. It predicts the next token given all previous tokens, appends it, and repeats. This simple recipe won because it is perfectly suited to GPUs during training and produces a clean probability for every token, which makes reinforcement learning post-training straightforward.
But the recipe carries three defects that are structural, not incidental:
For years the field accepted these defects as the price of doing business. Speculative decoding and KV caching shave the cost, but the fundamental loop stayed sequential. Diffusion language models change the loop itself.
Diffusion models already generate almost every image you have seen from an AI. The idea there is beautifully dumb: take a clean image, add a little noise, repeat until you have pure static. Then train a network to run the tape backward, removing a bit of noise at each step. Generation means starting from static and denoising your way to a picture.
Text is discrete. Words are not blurry, and you cannot add a fraction of a word. So researchers replaced Gaussian noise with something text-shaped: masking. The most influential formulation, popularized by the Kuleshov group and known as masked diffusion, is best understood as a generative BERT. You take clean text, hide a random fraction of tokens, and train a bidirectional transformer to fill in the blanks. Two differences from BERT matter. The masking rate is randomized across training, which turns out to make the model genuinely generative rather than just a fill-in-the-blanks classifier, and it comes with a principled training objective that closed most of the quality gap with autoregressive models.
Generation then works like this: start from a sequence that is entirely blanks, ask the model to fill in every blank, deliberately re-mask most of the sequence while keeping slightly more tokens fixed than last round, and repeat. The text assembles itself out of order, wherever the model is most confident.
Here is the sampling loop in pseudocode, so you can hold the whole algorithm in your head:
sequence = [MASK] * length # start: all blanks
for step in range(num_steps):
predictions = model(sequence) # fill in EVERY blank (a guess)
sequence = predictions # accept the full guess
keep = num_kept[step] # grows each round
sequence = remask_random( # re-noise, but keep the best
sequence, keep_count=keep)
If you know BERT, you already know 80 percent of this. The remaining 20 percent, the randomized masking schedule and the re-noising loop, is what turns a fill-in-the-blanks model into a generator that can write an entire passage in parallel and revise it mid-stream.
Plain masked diffusion had real problems: fixed-length output, no way to fix a token after unmasking it, and slow sampling relative to its potential. The current generation of models stacks four fixes, and the Kuleshov post traces each one. These are worth knowing by name, because they are the vocabulary the next year of model releases will use.
There is a fifth layer worth a sentence: post-training. Diffusion models complicate standard RL because estimating the likelihood of a full sampled sequence is expensive, so techniques like diffu-GRPO approximate it, and newer estimators have already pushed diffusion models to state-of-the-art results on logical and math reasoning benchmarks. The training recipe that made autoregressive models smart is being ported over.
So who is actually shipping this? Four names cover the field today.
That last detail deserves a . A model that speaks both languages, writing in parallel when you need throughput and sequentially when you want maximum reliability, tells you the industry does not see this as a religious war. It sees it as a per-request routing decision.
Here is the honest comparison, with the limits included, because 99 percent of autoregressive quality is not 100 percent, and diffusion models have not yet been trained at frontier scale.
My own takeaway from a week of reading: nothing here changes what I build this quarter, but it changes what I assume. I had quietly filed "LLMs generate left to right" next to physics constants. It is not a law. It is one algorithm, and another one now matches it on quality while beating it on speed.
The Kuleshov post ends with an argument I have not stopped thinking about. The transformer did not win because it was smarter than RNNs. It won because it was parallel, and parallelism is what let training scale. Since around 2024, gains from scaling pre-training have been flattening, and most new intelligence comes from post-training and inference-time compute, both of which are bottlenecked by how fast a model can generate, which today means a sequential algorithm. If diffusion makes inference fully parallel the way transformers made training parallel, the authors argue it could unlock a comparable jump. Their phrasing: diffusion may be to inference-time scaling what the transformer was to RNNs for pre-training scaling.
I would not bet the farm on any single research thesis. But I have read enough of these arcs to respect the shape of one: an incumbent approach with a structural speed limit, a challenger that removes the limit rather than optimizing around it, and open-weights releases making the challenger downloadable today. That is exactly what this looks like.
Here is what I would actually do with this:
I write about AI, backend engineering, and the tools I actually run, every week. Subscribe, it is free.
Have you tried a diffusion language model yet: Mercury, Gemma Diffusion, LLaDA, anything? Did the speed difference show up in your real workload, or was it a benchmark-only win? I am genuinely curious which way this breaks in practice.
Sources and further reading: