Vision Language Grounding: How AI Connects “Dog” to Pixels, and Where It Falls Apart

OpenAI researchers demonstrated in 2021 that CLIP, a vision language model, misclassifies a Granny Smith apple as an iPod when a paper label reading 'iPod' is attached, revealing that grounding in AI is statistical proximity in a learned space rather than conceptual understanding. The model, trained on 400 million image-caption pairs, aligns words and pixels through co-occurrence patterns, not symbolic checklists, causing failures when unusual poses, swapped attributes, or odd backgrounds shift representations away from their usual neighborhoods.

Picture an ordinary photo like this one. A vision language model will confidently caption it “a dog in a park,” and it will be right, almost every time 1 . The trouble starts at the edges of “almost.” In 2021, OpenAI researchers showed that printing a paper label reading “iPod” onto a Granny Smith apple caused their CLIP model to classify the apple as an iPod 2 . A related version used dollar signs drawn on a poodle, which shifted the model’s answer to “piggy bank” 2 . Neither attack required any optimization or special tooling, just pen and paper, because it exploits how CLIP encodes visual text as part of an image’s meaning 2 . That single demonstration is a good way into this whole topic. It shows that grounding is not conceptual understanding. It is proximity in a learned space, and that space can be nudged by anything that activates the right region, including text glued onto an object. Grounding, in the cognitive science sense, is how a word comes to refer to something real. When a child learns the word “dog,” they aren’t just memorizing a sound through repeated exposure, touch, and correction, they build a connection between that sound and the actual category of thing it refers to in the world. For a vision language model, grounding means something much narrower. The model never touches a dog or hears one bark. All it has is a word and a set of pixels, so grounding here just means: does the model’s internal representation of the word “dog” reliably line up with its internal representation of dog-shaped, dog-textured visual patterns? The mechanism behind that alignment is statistical, not symbolic, and this distinction explains nearly every failure covered later in this piece. A symbolic system would ground “dog” against something like a stored checklist: four legs, fur, tail, barks, and check an image against that checklist feature by feature. A statistical system, which is what these models actually use, has no checklist at all. There is only a mathematical space where the word “dog” and photos of dogs were pushed close together during training simply because they appeared together often enough. Nothing is verified or checked; it’s proximity, learned purely from co-occurrence patterns in the training data. That difference matters because a checklist generalizes gracefully: a dog in an odd pose is still a dog if you’re checking against a definition. Proximity in a learned space does not generalize as gracefully. Anything that shifts a word or image away from its usual neighborhood, an unusual pose, a swapped attribute, a negated sentence, an odd background they can break the association, even when a human would recognize the object instantly. CLIP is built as a dual encoder. That means there are two separate neural networks working side by side. One is a Vision Transformer, which reads raw pixels. The other is a Transformer, which reads the caption text. Each of these two networks processes its own input completely on its own, without ever looking at what the other one is doing. What makes them usable together is that both networks are designed to output a vector, a plain list of numbers, and both vectors are the exact same length. Because they are the same length, they can be directly compared to each other, even though one came from a picture and the other came from a sentence. This shared, comparable output space is what “project into the same latent space” means 1 . To train this system, researchers used 400 million image and caption pairs collected from the internet 1 . For every single pair, the training process nudges the image’s vector and its correct caption’s vector closer together in that shared space. At the same time, it pushes that same image’s vector away from every other caption that does not belong to it. This is done in a very specific way. For every batch of examples during training, the model builds a large grid of similarity scores, comparing every image in the batch against every caption in the batch. It then applies a loss function called cross entropy, which penalizes the model whenever a wrong pairing scores higher than the correct pairing. This happens in both directions at once, meaning the model is checked on matching images to captions, and separately on matching captions back to images 1 . Put simply, the training task boils down to this. Given one image, and a large pile of caption candidates where only one is correct, the model has to correctly pick out the one true caption 1 . It gets rewarded when it guesses right and corrected when it guesses wrong, and it does this across hundreds of millions of examples. That repeated correction, done at massive scale, is the entire reason the model ends up placing the word “dog” near photos of actual dogs. Nobody ever gave it a definition of what a dog is. Here is a diagram of that full pipeline, showing both encoders and how their outputs land in the same shared space. The loss function that drives this training is called InfoNCE. Below is that same mechanism written out as runnable code, so you can see there is no hidden step: python import torchimport torch.nn.functional as Fdef contrastive loss image embeds, text embeds, temperature=0.07 : """ image embeds, text embeds: batch size, dim , L2-normalized. Row i of each tensor is assumed to be a true match. """ logits = image embeds @ text embeds.T / temperature labels = torch.arange logits.shape 0 , device=logits.device loss i2t = F.cross entropy logits, labels loss t2i = F.cross entropy logits.T, labels return loss i2t + loss t2i / 2 Reading through this code line by line, image embeds @ text embeds.T is the similarity grid described above. Dividing by a temperature value simply sharpens or softens how confident the scores are. The two cross entropy calls are the two directions of checking mentioned earlier, image to text and text to image. Averaging them at the end gives one final loss number the model can learn from. Finally, here is what grounding looks like once a model has already finished training, using Hugging Face’s transformers library: python from transformers import CLIPModel, CLIPProcessorfrom PIL import Imageimport torchmodel = CLIPModel.from pretrained "openai/clip-vit-base-patch32" processor = CLIPProcessor.from pretrained "openai/clip-vit-base-patch32" image = Image.open "dog photo.jpg" captions = "a photo of a dog", "a photo of a cat", "a photo of a car" inputs = processor text=captions, images=image, return tensors="pt", padding=True with torch.no grad : probs = model inputs .logits per image.softmax dim=1 for c, p in zip captions, probs 0 : print f"{c}: {p.item :.3f}" This snippet does not train anything. It loads a model that has already learned from those 400 million examples, feeds it one real photo along with three candidate captions, and asks it to score how well each caption matches. If everything worked as described above, the caption “a photo of a dog” should come back with the highest score whenever the photo actually shows a dog. Running this yourself turns the whole explanation from something you read into something you can watch happen. A single pooled vector can tell you whether an image and a caption match overall, but it cannot tell you where inside the image the match actually is. If a photo shows a dog in one corner and a person in the middle, one vector for the whole image has no way to point specifically at the dog. Three approaches address this. RegionCLIP compares small pieces of an image to text, instead of only comparing the whole image to a whole caption. It first generates labels for individual regions of an image, then trains the model to match those regions against pieces of text, before this is transferred into a full detection setup 9 . OWL-ViT starts from an already trained CLIP style model. It removes the step that pools the whole image into one single vector, and instead attaches small heads directly onto the individual output tokens from the image encoder, where each token corresponds to a small patch of the image. At the time of use, a text query is compared against those tokens to find the matching patch, which is what lets it detect objects described by any free text phrase, not just a fixed list of categories 8 . Grounding DINO fuses the two modalities earlier in the process. Image tokens attend to each other, and at the same time attend to the text tokens, so both sides share a grounded understanding of the scene before the model proposes any specific region as its answer 10 . All three approaches rely on the same underlying comparison, just applied to small image patches instead of a whole image. A word is compared against every patch, each patch gets a weight showing how strongly it relates to that word, and the patch with the highest weight is where the model is, functionally, pointing: python import torch.nn.functional as Fdef patch attention word embed, patch embeds : """word embed: dim, patch embeds: num patches, dim """ scores = patch embeds @ word embed return F.softmax scores, dim=0 weight per patch, sums to 1 patch embeds @ word embed produces one similarity score per patch. Passing those scores through softmax turns them into weights that add up to one, so the patch with the highest weight is the model's best guess at where the word is grounded in the image. Attribute binding means correctly matching a property, like a color, to the specific object it belongs to. This is where vision language models frequently break down. Research has found that CLIP often behaves like what researchers call a bag of words model. This means the model notices which words and concepts are present in a caption, but does not reliably track which word belongs to which object in the scene. For example, given an image showing an orange square next to a blue triangle, CLIP will often score that image just as highly against the caption describing a blue square and an orange triangle, even though the colors have been swapped between the shapes 3 . This failure was studied carefully through something called the ARO benchmark, short for Attribution, Relation, and Order. It was built specifically to test whether vision language models actually understand attributes, relationships between objects, and the order words appear in a sentence. The benchmark included more than 50,000 test cases, and the results showed that even state of the art models regularly make mistakes when trying to correctly link an object to its attribute 3 . One explanation researchers have proposed for why this happens comes down to how these models are trained. During training, a model is shown a batch of images and captions, and it is taught to prefer the correct pairing over the incorrect ones in that same batch. If the incorrect pairings in a batch are too easy to rule out, meaning they do not closely resemble the correct answer, the model can still succeed at the training task without ever learning to carefully track which attribute belongs to which object. In other words, without enough close, tricky wrong answers to learn from, the model can take a shortcut and still score well 3 . Test it yourself: captions = "a red cube on top of a blue sphere", "a blue cube on top of a red sphere" inputs = processor text=captions, images=image, return tensors="pt", padding=True probs = model inputs .logits per image.softmax dim=1 print probs similar scores against one image = the binding failure If the two probabilities printed here come out close to each other, that confirms the model cannot reliably tell which caption actually matches the colors in your image. A spurious correlation happens when a model learns to rely on something that is not actually the object it is supposed to be recognizing, simply because that other thing tended to appear alongside the object during training. This problem is not unique to dogs, and it is not unique to vision language models either. One clear example involves keyboards. If keyboards in a training dataset are almost always photographed with a monitor sitting nearby, a model can end up learning to detect the presence of a monitor as a stand in for detecting the keyboard itself. This works fine as long as keyboards and monitors keep appearing together, but it fails as soon as the two appear separately, since the model never actually learned what a keyboard looks like on its own 12 . CLIP shows this same pattern. Researchers studied CLIP’s predictions by grouping them according to the background present in each photo. They found strong, consistent associations between certain backgrounds and certain predicted categories. As one example, images with railway tracks in the background strongly pushed CLIP toward predicting the object was a train, even in cases where the actual object had nothing to do with trains. This likely reflects how frequently railway tracks and trains appeared together during training 13 . This issue also extends beyond simple classifiers into more advanced multimodal systems. Research on multimodal large language models found that the vision encoder component on its own already carries these same spurious biases, meaning the problem is not something introduced only when vision and language are combined. The research also found that these misleading cues can directly cause the model to hallucinate objects that are not actually present in the image 14 . Counting requires a model to track individual, separate instances of an object rather than just recognizing that the object category is present somewhere in the image. This turns out to be a genuinely difficult task for these models. CountBench was created specifically to measure how well vision language models can count objects, after researchers observed that CLIP style models consistently struggled with this task 6 . A more recent and more tightly controlled benchmark, called VLMCountBench, pushed this testing further using simple geometric shapes instead of complex real world objects, to remove other sources of confusion. This study found substantial failures in counting whenever two or more different shape types appeared together in the same image. These failures showed up even when the total number of objects was small and the scene was visually simple. Performance also got worse when researchers made small changes to color, size, or how the counting question itself was phrased, showing that the models were not applying a stable, reliable counting process 7 . Negation means understanding when a sentence is saying something is absent, rather than present. This is a specific and well documented weakness in vision language models. Recent evaluation work found that these models show a strong affirmation bias, meaning they tend to treat a caption like “a dog” and a caption like “no dog” as nearly the same thing when comparing them against an image, largely ignoring the negation 4 . A separate research paper describes the same issue directly, explaining that CLIP struggles to understand negation and often produces very similar embeddings for affirmative and negated descriptions of the same scene. As a concrete example, the caption “no dog” ends up scoring a strong match against images that clearly do contain a dog, which is the opposite of what should happen 4 . This weakness is not limited to English either. A benchmark built to test negation understanding across seven different languages found that standard CLIP performed at or below random chance specifically on languages that do not use the Latin alphabet, showing the problem can be even more severe depending on the language involved 15 . This failure mode does not come from a single dedicated benchmark the way the previous ones do. Instead, it follows logically from how these models are trained in the first place. Since the model’s internal space is shaped entirely by the examples it was trained on, any part of that space corresponding to rarely seen visual situations will be less reliably organized. This includes unusual poses, objects that are heavily blocked from view, or breeds and variations that were underrepresented in the training data. Because this pattern connects back to the same root cause behind the other failures, it acts as a kind of thread running underneath all of them, rather than being its own separate, isolated issue. Object hallucination refers to a model describing something in an image that is not actually there. The standard benchmark used to study this is called POPE. The researchers behind it conducted the first systematic study of this problem in large vision language models, and found that these models frequently suffer from serious object hallucination, meaning they describe objects that do not match what is actually shown in the image 5 . The way POPE tests for this is straightforward. It asks the model a simple yes or no question, such as whether a particular object is present in the image. It does this across three different settings. In the random setting, the object asked about is chosen at random. In the popular setting, the object is one that appears frequently across the training data in general. In the adversarial setting, the object asked about is one that closely and commonly co-occurs with objects that actually are present in the image, making it a harder and more deceptive test 5 . Follow up research building on this benchmark found that models are prone to hallucinating whether an object exists at all, and this problem becomes even more pronounced when testing whether the model correctly identifies specific, fine-grained attributes of objects that are genuinely present 5 . Covered in the hook, worth restating precisely: this was documented in the paper studying adversarial attacks on multimodal neurons, building on OpenAI’s original typographic attack findings 2 . Defenses exist too: researchers have proposed a learned output transformation, fine-tuning on synthetic typographic attack images combined with weight interpolation, and inserting a learned “defense prefix” token before the class name, as three separate mitigation strategies 16 . Every failure above traces back to one property: the embedding space encodes statistical co-occurrence, not compositional structure. A pooled vector is a summary of features that tended to appear together, with no explicit mechanism for attribute binding, negation, or counting, and no guarantee that rare combinations are represented as reliably as common ones. Grounding is geometric, not perceptual, and every study cited above shows that geometry failing in specific, reproducible ways. That’s good news for anyone building on these systems: the failure modes are catalogued, benchmarked, and under active study, not mysterious. What does “grounding” mean for an AI model, in simple terms? It means connecting a word to the right thing in an image. For a person, this connection comes from lived experience. For a model, it comes from proximity in a mathematical space, meaning the model places the word “dog” and photos of dogs close together because they appeared together often enough during training, not because it understands what a dog actually is. Is CLIP the only model that works this way? CLIP is the most well known example and the one this piece focuses on, but the same dual encoder, contrastive learning approach shows up across many later systems, including OWL-ViT and RegionCLIP, both of which build directly on CLIP style pretraining before adding detection capabilities 8 9 . Why does the apple and iPod experiment work? Because the model was trained to associate visible text with meaning, since captions on the internet often describe or label what is shown. Printing “iPod” on an apple activates that same text recognizing part of the model strongly enough to override the actual visual features of the fruit 2 . Can these failures be fixed by just training on more data? Only partially. More data reduces how rare unusual combinations are, which helps with issues like rare poses or uncommon breeds. It does not fix problems like attribute binding or negation, since those come from the model never being forced to learn compositional structure in the first place, not from a lack of examples 3 4 . Which failure mode is the most well studied? Object hallucination is probably the most actively benchmarked, largely because of POPE, which gives a simple, repeatable way to test whether a model reports objects that are not actually present in an image 5 . Do these problems affect every vision language model equally? No. Different architectures handle grounding differently, and some approaches, like region level contrastive training or early fusion in Grounding DINO, specifically target certain weaknesses such as poor localization 9 10 . That said, none of the published fixes fully close every gap discussed in this piece. Is negation a bigger problem in some languages than others? Yes. A benchmark testing negation across seven languages found that standard CLIP performed at or below chance specifically on languages that do not use the Latin alphabet, showing the problem is not evenly distributed across languages 15 . Can I test any of these failures myself without training a model? Yes. Every code snippet in this piece uses an already trained, publicly available CLIP model through the Hugging Face transformers library. You can run the attribute binding test, the counting test, or the negation test yourself with just a few lines of Python and any photo of your choice. Where can I read the actual research behind these claims? Every factual claim in this piece is tied to a numbered reference in the References section at the end, linking back to the original paper or benchmark it came from. 1 Radford, A. et al. 2021 . Learning Transferable Visual Models From Natural Language Supervision CLIP . OpenAI. https://openai.com/index/clip/ https://openai.com/index/clip/ 2 Goh, G. et al. 2021 . Multimodal Neurons in Artificial Neural Networks . Distill. 3 Yuksekgonul, M., Bianchi, F., Kalluri, P., Jurafsky, D., & Zou, J. 2023 . When and Why Vision-Language Models Behave Like Bags-of-Words, and What to Do About It? ICLR. https://arxiv.org/abs/2210.01936 https://arxiv.org/abs/2210.01936 4 Alhamoud, K. et al. 2025 . Vision-Language Models Do Not Understand Negation . CVPR. 5 Li, Y. et al. 2023 . Evaluating Object Hallucination in Large Vision-Language Models POPE . https://arxiv.org/abs/2305.10355 https://arxiv.org/abs/2305.10355 6 Paiss, R., Ephrat, A., Tov, O. et al. 2023 . Teaching CLIP to Count to Ten . ICCV. 7 Yu, S. et al. 2025 . Your Vision-Language Model Can’t Even Count to 20: Exposing the Failures of VLMs in Compositional Counting . https://arxiv.org/abs/2510.04401 https://arxiv.org/abs/2510.04401 8 Minderer, M. et al. 2022 . Simple Open-Vocabulary Object Detection with Vision Transformers OWL-ViT . ECCV. 9 Zhong, Y. et al. 2022 . RegionCLIP: Region-Based Language-Image Pretraining . CVPR. 10 Liu, S. et al. 2023 . Grounding DINO: Marrying DINO with Grounded Pre-Training for Open-Set Object Detection . 11 Kim, D., Angelova, A., & Kuo, W. Region-Aware Pretraining for Open-Vocabulary Object Detection with Vision Transformers RO-ViT . https://arxiv.org/abs/2305.07011 https://arxiv.org/abs/2305.07011 12 Trustworthy Machine Learning — context bias, keyboard/monitor example. https://arxiv.org/pdf/2310.08215 https://arxiv.org/pdf/2310.08215 13 Concept Regions Matter: Benchmarking CLIP with a New Cluster-Importance Approach . https://arxiv.org/pdf/2511.12978 https://arxiv.org/pdf/2511.12978 14 Seeing What’s Not There: Spurious Correlation in Multimodal LLMs . https://arxiv.org/html/2503.08884v1 https://arxiv.org/html/2503.08884v1 15 Moraitaki, C. et al. Disparities in Negation Understanding Across Languages in Vision-Language Models . https://arxiv.org/pdf/2604.18942 https://arxiv.org/pdf/2604.18942 16 Materzyńska, J. et al. — learned output transformation defense against typographic attacks; Azuma, R. & Matsui, Y. — Defense-Prefixes. Referenced via https://arxiv.org/pdf/2504.08729 https://arxiv.org/pdf/2504.08729 Vision Language Grounding: How AI Connects “Dog” to Pixels, and Where It Falls Apart https://pub.towardsai.net/vision-language-grounding-how-ai-connects-dog-to-pixels-and-where-it-falls-apart-56d4f91ec671 was originally published in Towards AI https://pub.towardsai.net on Medium, where people are continuing the conversation by highlighting and responding to this story.