How do image and video generators work, and what is diffusion?

Image and video generators are the type that turns a text prompt into a picture. They are a useful contrast to language models, because they make a different bet about how to produce their output. Most of them are diffusion models, and diffusion is a genuinely different objective from next-token prediction.

The idea is to learn the reverse of a corruption process. During training, the method takes a real image and adds noise to it in small steps until nothing is left but static. The model learns to undo that, predicting how to remove a little noise at each step^[1]. To generate, you run only the reverse direction: start from pure random noise and let the trained network denoise it again and again, until a coherent image emerges from what began as static^[3]. The prompt steers the denoising so the result matches what you asked for.

That is the split worth holding onto. A language model is autoregressive: it builds its output one token at a time, left to right, each step conditioned on what came before. A diffusion model works on the whole image at once and refines all of it together across many passes. Same goal of generating something new, opposite mechanics.

Early diffusion ran directly on pixels and was slow. The shift that made it practical was latent diffusion, which compresses an image into a smaller representation, runs the denoising there, and decodes back to pixels at the end. This cut the cost enough to train high-resolution generators on ordinary hardware^[2], and it underpins much of the image generation in wide use. The same approach extends to video, where the model denoises frames that have to stay consistent over time, which is why video generation is harder and more expensive than a single still.