Hello, I'm Rijul, and I'm building LiveReview — a blast-radius aware AI code review built for your business-critical systems. Star us to help devs discover the project, give it a try, and share your feedback to help improve the product.
Do you interact with AI models where you put in an image and then it processes it?
Do you wonder what these models are and what is happening behind the scenes?
This is called a VLM, or Vision-Language Model.
A VLM combines vision and language capabilities so that it can process visual information along with text.
A simple way to think about it is:
LLM → Works primarily with text
VLM → Works with images and text
This is a simplified mental model, since VLM architectures can vary. Under the hood, a VLM often combines a vision encoder, a connector or projector, and a language model.
So, how does a VLM actually "see" an image?
An LLM works with tokens.
But an image is not naturally made up of language tokens.
So we need a way to convert the visual information in an image into something the language model can work with.
A simplified view looks like this:
Image
↓
Vision Encoder
↓
Visual Representations
↓
Vision-Language Connector
↓
Language Model
↓
Text Response
Let's break this down.
The first important component is the Vision Encoder.
Its job is to process the image and turn the visual information into numerical representations that the rest of the model can work with.
Depending on the vision architecture, the image may be divided into smaller regions or patches.
For example:
Each part is processed by the vision encoder, which produces visual representations.
These representations don't simply mean "this patch is a dog" or "this patch is blue."
The vision encoder builds representations that capture visual patterns and context, which can help the model understand things such as shapes, objects, colors, text, spatial relationships, and other visual information.
The exact way this happens depends on the vision encoder architecture. Patch-based processing is common, but it is not universal to every vision system.
The result is essentially a representation of the image that can be passed toward the language model.
Now that we have visual representations, we can talk about two important capabilities: perception and grounding.
Perception is about understanding the visual content.
It is not just:
"What objects are in this image?"
It can also involve understanding relationships between objects, actions, scenes, and the overall visual context.
For example, given an image of a person riding a bicycle, the model may understand:
Person
↓
riding
↓
Bicycle
So perception is about building an understanding of what is happening in the image.
Grounding is about connecting language to specific parts of the visual input.
For example, if we ask:
"Where is the dog?"
The model could associate the word dog with a particular region of the image.
A simplified representation could be:
{
"object": "dog",
"location": "center of the image"
}
In more technical systems, grounding can involve associating language with regions, objects, points, or other parts of an image. Bounding boxes are one common way of representing this, but they are not the only form of grounding.
This becomes especially useful when we want an AI system to not only identify something, but also connect that understanding to where or what it refers to in the image.
You may also hear another term:
Multimodal LLM
So what is the difference?
The terminology isn't perfectly standardized, but a useful way to think about it is:
VLM usually emphasizes models that connect vision and language.
Examples include models such as:
Multimodal LLM is a broader term often used for language models that can work with multiple types of inputs or outputs, such as:
Text
Images
Audio
Video
So, conceptually, you can think of VLMs as one part of the broader multimodal AI space.
The exact terminology can vary depending on the model and how its creators describe it, so these categories should be treated as a useful mental model rather than strict technical boundaries.
So, the basic idea behind a VLM is:
Image
↓
Vision Encoder
↓
Visual Representations
↓
Connector / Projector
↓
Language Model
↓
Response
The vision side processes the image and produces visual representations.
The connector helps make those representations usable by the language model.
The language model then uses those visual representations together with the text input to generate a response.
VLMs become especially useful when text alone isn't enough to express what we want an AI system to understand.
For example, instead of describing a UI design, diagram, screenshot, chart, or photograph entirely through text, we can simply give the model the image.
In that sense, VLMs give AI systems a way to work with visual information alongside language, making it possible to interact with information that previously had to be described using words.
So that's it about this article, see you on the next one!
Your team's attention is limited, and the deluge of AI-generated code is making it harder to keep production stable while also shipping at high velocity.
I'm building LiveReview, a blast-radius aware AI code review built for your business-critical systems.
Instead of presenting every diff with equal emphasis, LiveReview scores each change by blast radius — how far its impact reaches through your call graph — so you can focus attention where it actually matters.
Spend code review effort where business risk is highest — not spread evenly across every diff.
Try LiveReview on your codebase: