A model that only ever read text now looks at a photo of your fridge and suggests a recipe. Nothing about "predicting the next token" obviously explains how it can see. The bridge is simpler and more elegant than it looks.
Multimodal AI — models that handle images, audio, and text together — feels like a different kind of technology. It isn't. It's the same machinery with one clever addition. Here's the idea.
A language model reasons over embeddings — vectors that capture meaning — derived from text tokens. The entire breakthrough of multimodal models is this: you can turn an image (or audio) into vectors in that same space.
An image gets sliced into patches, and each patch is encoded into a vector, much as a sentence is split into tokens and embedded. Once an image is a sequence of vectors living in a shared space with text, the model's attention mechanism can treat words and image-patches as one combined sequence — letting a word attend to a region of an image and vice versa. "See" isn't a new faculty bolted on; it's the old faculty fed a new kind of input that's been translated into the language the model already speaks: vectors.
That's the whole conceptual leap. Different senses, one representational space. It's the same "meaning as geometry" principle that powers embeddings and semantic search, extended past text — a unifying idea I find genuinely elegant and lean on across the systems I build.
Once images and text live in the same space, capabilities fall out almost for free:
The same recipe extends to audio, video, and beyond. Encode the modality into the shared vector space and the model can reason across it.
The elegance hides real difficulty. Aligning the spaces well — so an image and its description truly land near each other — takes enormous, carefully-paired training data. And the failure modes compound: a multimodal model can misread an image and then confidently describe what it thinks it saw, layering a perception error under a hallucination. Trusting the output means accounting for both.
Multimodal is where a lot of the field is heading, because the world isn't made of text. Systems that can perceive and reason across images, sound, and language unlock everything from document understanding to real-world visual tasks. And once you see it as "translate every modality into one shared space of meaning," it stops being mysterious and starts being buildable.
The senses are different; the representation is one. That's the whole idea. More of how I think about it at www.divyakush.com.
Divyakush Punjabi · Full-Stack & AI Engineer