Types of models
What are the main kinds of AI model, and what separates them?
AI models split into a handful of types: language models, multimodal models, image and video generators, speech and audio models, and embedding models. Two questions sort them. What does it read and produce, and what was it trained to optimise. Get those two clear and the families fall into place.
It helps to have a map before you learn the family names. The models you hear about sort into a few types, and almost all of the sorting comes down to two questions. What does the model take in and give back, and what was it trained to do.
The first question is about modality, meaning the kind of data involved. A language model reads text and writes text. A multimodal model can also take in images, audio, or video. An image generator turns a text prompt into a picture. A speech model moves between audio and text in either direction. An embedding model reads something and returns a list of numbers rather than anything a person reads directly. Modality alone gets you most of the way to naming a type.
The second question is about objective, the thing the model was trained to optimise. Two models can share a modality and still be different kinds of thing. A language model and an embedding model both read text, but one is trained to predict the next token and produce more text[1], while the other is trained to place meaning in a vector space so similar things land near each other[3]. The same split shows up in images. Some image models generate by predicting pieces in sequence, the way a language model does, while diffusion models generate by starting from noise and removing it step by step[2]. Same output, different objective, different type.
The rest of this topic walks each type in turn: large language models, multimodal models, image and diffusion models, speech and audio models, and embedding models. Hold the two axes in mind as you read. When a new model appears and you are not sure what it is, ask what it reads, what it produces, and what it was trained to optimise. The answer usually tells you the type before anyone tells you the name.
Two types on this map are easy to conflate and worth separating early. A multimodal model reads more than text but still mostly writes text back — it is a language model with extra senses. An image or video model is a different objective altogether: its whole job is producing pixels, and reading a text prompt is just the steering input. Modality tells you what a model can perceive; objective tells you what it is for. You need both answers, not just one, to know which type you are actually looking at.
References
- The Illustrated GPT-2 (Visualizing Transformer Language Models) — Jay Alammar
- Denoising Diffusion Probabilistic Models — Ho, Jain & Abbeel (arXiv)
- Embeddings — OpenAI
Common questions
- Is every modern AI model a transformer?
- Most well-known ones are, including language and multimodal models, but not all. Many image generators are diffusion models with a different internal design, so the model type is about what it does, not just the architecture inside it.
- Why group models by modality and objective rather than by company?
- Company and product names change every few months. What a model reads, produces, and was trained to optimise changes far more slowly, so it is the steadier way to understand the field.