Reading into VLM hallucinations using the Jacobian lens

A researcher used Anthropic's J-lens method to analyze LLaVA-1.5-7B's internal state during visual question answering, finding that the model's internal workspace accurately registers object presence or absence even when the model hallucinates a 'yes' answer. The hallucination appears to stem from the yes/no question format and a failure to leverage known visual information, as switching to a forced-choice format yields correct answers.

Reading and editing the visual workspace of a vision-language model. Vision-language models hallucinate: ask one whether some object is in a picture and it will happily say yes whether it's there or not. Why is this? Using Anthropic's recent J-lens method https://www.anthropic.com/research/global-workspace I found that LLaVA-1.5-7B's internal state seems to register that the object is absent. The exact same evidence that the yes/no question ignores produces almost perfect answers when posed as a choice instead "is this a lamp or a dog?" . Perhaps then the hallucination can be shown to be a product of the question format and a failure to pull up things that the model already knows. The knowledge of the image turns out to be readable and editable inside the model using Anthropic's J-lens. Scope: one model LLaVA-1.5-7B , 13 categories, 39 core images, greedy decoding. Every number here comes out of cached data in the repo linked at the end. Show the model a photo with no lamp in it. Ask: "Is there a lamp in this image? Answer yes or no." . Do it for 39 different lamp-less images and it says yes 39 times out of 39. It also says yes 39/39 when the lamp really is there. There is a known yes-bias problem here https://arxiv.org/pdf/2412.20622v1 and here https://arxiv.org/pdf/2605.04641 with this family of models. Using the Jacobian lens, we can read the model's internal state during the hallucination. It seems like if we rely only on the J-space the model knows what is present and what is not. The Jacobian lens is a way to turn a layer's raw activations into scores over the model's own vocabulary. In order to ask "how readable is the word lamp right here?", point it at the image tokens the ~576 positions where the picture lives inside the prompt and rank where the true word lands out of 32,000. Objects genuinely in the picture sit at a median rank of 16 . The suggested-but-absent lamp sits at 111, roughly an order of magnitude weaker. An absent object that wasn't asked about sits at 430 . A plain logit lens shows the same separation seen ~11 vs absent 282/3142 . This is a fact about LLaVA, not about the Jacobian lens. Neo et al. showed https://arxiv.org/abs/2410.07149 object identity is decodable at these positions with a plain logit lens; the J-lens differs in that it transports the activation through the average Jacobian rather than unembedding it directly. That jump from 430 down to 111 when the question named the word shows us that naming a word in the prompt makes it a little more readable at the image tokens regardless of the picture's content plausibly the question word echoing in through attention . Even when the absent object is included in the prompt, it stays about an order of magnitude weaker in the workspace than anything the model really did see. There is some distinction here in the workspace between 'asked in the prompt' and 'actually seen in the image' that the LLM is not able to make when answering. We are able to track when the model starts to lean towards answering "yes". At the position where the next word is chosen, you can measure how strongly the model leans Yes vs No at each layer. The lean is flat early, starts climbing around layer 13, peaks around layer 20, and settles positive at the output. The curve for asking about a present and an absent object are almost the same. Since they agree on the verdict, maybe this is to be expected as the image stays the same and the prompt changes by only one word. The object being visible in the left graph in the workspace makes little difference to the result of the model here. Left: the internal workspace, present being separated from absent. Right: the answer being decided Is it just that the model can't use its visual evidence for this kind of question? I don't think so. Same 39 images and word pairs with a different question: "Which of these is in the image: a lamp or a dog?" Asked both ways round so it can't just pick the first option gives an accuracy of 38/39 and 37/39 . So the evidence is present and readable the whole time. A question that names both options and forces a choice uses it well. A yes/no question about one option defaults to agreement. Note that the question was given before the image in these tests. At least here, hallucination doesn't seem to be a perception failure, and isn't even about the evidence being overridden. The correct signal is inside the model linearly readable, but at natural evidence levels it's swamped out by the model's yes-bias. In cases of hallucination, the truly present objects can be read from the J-space instead of the output tokens. The natural next step is to interfere with what the model sees in each image. Luckily we can edit the workspace. The lens gives you a direction for "dog" and a direction for "cat"; inside the model you can swap how much of each is present, at some dosage α α=1 is an exact exchange . Editing the words output from the model gets us in a loop "cat cat cat..." . Editing only the input positions prompt, image tokens works nicely. The natural ancestor to these tests is again Neo et al. https://arxiv.org/abs/2410.07149 . Increasing α predictably and smoothly changes the model's preference for what it saw to the imposter. There is a tie at α=1, and the full flip to the imposter happens past 1 α ≈ 1.25–1.5 . Swapping two unrelated words as in, irrelevant to the real image at the same strength doesn't move anything. Some image descriptions from the model before and after a word swap: The Dalmatian swap: swap dog→cat and the broad category of animal flips, but the spots it saw survive the edit Swapping cat into dogs of other breeds keeps attributes like the dogs smile or tongue. The injected concept also drags in its own baggage. Swapping "car" with "plane" brings a sky with it that was not in the real image. With two objects side by side, each has its own handle: swap the dog and the cup next to it is left untouched. Last one. The vision encoder here is a frozen CLIP, 24 layers of patch embeddings with no word vocab and no unembedding. Perhaps it can hold something word-shaped on its own, before the language model gets involved? I fit a J-lens for each encoder layer. The CLIP lens was averaged over 144 images and calibrated per word. I am hesitant to frame this as a genuine, category-general word channel, but from about encoder layer 10 up it seems something like a J-lens workspace is able to be found. Categories the lens had never seen an image of still read near the top. Two things are needed before any signal shows up. First, read the patch-averaged activation, not individual patches. Second, calibrate per word. Calibration here is subtracting each candidate word's mean score across images because some words score high on every image and bury the signal this is the same correction the original paper needed . With both, a seemingly category-general channel appears from about layer 10. The layer-22 lens reads layer-8 activations nearly as well as layer-8's own lens does, so this is one shared channel decoding a signal that's linearly present early, not something computed at layer 10. Of course a far weaker result than the Anthropic language-model side. This is picking the right word out of 45 after calibration compared to 32,000 in vocab for the LLM. Gandelsman et al. https://arxiv.org/abs/2310.05916 showed CLIP's internal contributions are text-interpretable, but via CLIP's own shared image-text embedding space, decomposing the direct effects on the final representation. This post reads intermediate CLIP layers through a Jacobian into a different model's vocabulary. So "CLIP internals are word-readable" is known, this specific readout mechanism is not. Most of this is known already. These models hallucinate, the phrasing of the question changes how often they do, as found in work like POPE https://arxiv.org/pdf/2305.10355 . Of course so is the ability to decode more from a model's activations than it says out loud. The coordinate swap is from the Anthropic paper https://www.anthropic.com/research/global-workspace . The closest work I could find is Neo et al. https://arxiv.org/abs/2410.07149 , the differences between this post and the paper is the cross-boundary framing and the three-way present/asked-absent/unasked-absent comparison. As far as I can tell the Jacobian-lens method hadn't been pushed across a model's vision-language boundary before. I was happy to see information from the vision encoder susceptible to the same method and readable throughout the VLM. Everywhere that content is read at the language-model layers the lamp readouts, the forced choice , a plain logit lens gives essentially the same answer, those results don't depend on the Jacobian lens. It only does something a logit lens can't when reading the frozen vision encoder internals. In CLIP a naive projection of an intermediate activation sits at chance, and only the measured Jacobian recovers the word. The editing results use the lens directions rather than reading them off, so I haven't checked whether a logit lens would swap as cleanly. The repo: https://github.com/jude-sph/J-lens-Vision https://github.com/jude-sph/J-lens-Vision Code and results were written with AI-assistance.