How does a multimodal model take in images and audio, not just text?

A multimodal model is one that takes in more than one kind of data. The most common sort is a vision-language model: you give it an image and some text, and it answers in text. You can hand it a screenshot and ask what a button does, show it a photo and ask what is in it, or paste a chart and ask it to read off the trend. Recent frontier models go further and accept audio and video as well, mapping all of it into one space the model can reason over together^[2].

The trick that makes this work is that images and audio are turned into the same currency the model already trades in. A language model thinks in tokens, the numeric pieces it learned text from. A multimodal model adds an encoder that converts an image, or a slice of audio, into vectors that live in that same space. An influential step here was CLIP, which trained on 400 million image and caption pairs so that a picture and the words describing it land near each other^[1]. Once an image is represented in a form the language part understands, the model can attend to it exactly as it attends to words, and reason across the two.

So the type is best understood as a language model with extra senses bolted on, sharing one set of weights. That sharing is the point. Because the picture and the prose sit in the same representation, the model can answer questions that need both at once, like reading the text in a screenshot and acting on what it says.

Two limits are worth carrying. First, more input modalities do not always mean output modalities: many multimodal models read images and audio but still mainly write text. An any-to-any model that also produces images or speech is a smaller and newer set. Second, the same failure modes as a plain language model still apply. The model can misread a blurry image or describe something that is not there with full confidence, so vision does not remove the need to check.