Model architectures
How does a diffusion model differ from an autoregressive one?
Diffusion models generate by reversing a noising process: they start from noise and denoise toward a finished output over many steps. That is the dominant design for image generation. Newer diffusion-LLMs apply the idea to text, refining a whole sequence in parallel rather than left to right, which can be much faster than autoregressive decoding.
Most language models you have met are autoregressive: they write one token, then the next, each conditioned on what came before, left to right. A diffusion model works the other way around. Training corrupts a real example by adding noise in many small steps until it is pure noise, and the model learns to run that process in reverse. At generation time it starts from noise and denoises toward a finished output over a series of refinement steps, sharpening the whole thing at once rather than building it piece by piece. This is the design behind nearly all modern image generators, and the original formulation showed it could match or beat the image quality of the methods it replaced.[1]
The interesting recent move is bringing diffusion to text. Language is discrete, so text diffusion models replace continuous noise with masking: the forward process hides more and more tokens, and the model learns to fill the masked positions back in. LLaDA, an 8-billion-parameter diffusion language model trained from scratch, showed this can compete with a similarly sized autoregressive model on in-context learning and instruction following, and it even handled some tasks, like completing a poem in reverse, that trip up left-to-right models.[2] Commercial systems have followed: Inception Labs’ Mercury Coder, a diffusion language model built for speed, reports 1109 tokens per second on an H100 for its Mini size and 737 for Small — up to roughly 10x the throughput of speed-tuned autoregressive models.[3]
The contrast with autoregressive generation is the whole point. Writing left to right means each token waits for the one before it, which caps how fast you can go. Diffusion refines many positions in parallel, so it can be much faster, and because it can revisit and edit the whole sequence, it can correct earlier choices rather than being stuck with them. The cost is that it needs several denoising passes per output and is a less proven recipe for language than the autoregressive approach that almost every large model still uses.
References
- Denoising Diffusion Probabilistic Models (arXiv:2006.11239) — Ho, Jain & Abbeel, UC Berkeley
- Large Language Diffusion Models (LLaDA) (arXiv:2502.09992) — Nie et al.
- Mercury: Ultra-Fast Language Models Based on Diffusion (arXiv:2506.17298) — Inception Labs
Common questions
- Why are most chatbots autoregressive instead of diffusion?
- Autoregressive text generation, writing one token at a time, has been the proven recipe for language and is what almost every large language model uses. Diffusion text models are newer; they trade the familiar left-to-right approach for parallel refinement, which can be faster but is still less established for language.
- Are diffusion language models real, or just research?
- Both. They started in research with open models like LLaDA, and there are now commercial diffusion language models such as Inception Labs' Mercury, which is built to generate text far faster than typical autoregressive models.