{"slug": "reading-into-vlm-hallucinations-using-the-jacobian-lens", "title": "Reading into VLM hallucinations using the Jacobian lens", "summary": "A researcher used Anthropic's J-lens method to analyze LLaVA-1.5-7B's internal state during visual question answering, finding that the model's internal workspace accurately registers object presence or absence even when the model hallucinates a 'yes' answer. The hallucination appears to stem from the yes/no question format and a failure to leverage known visual information, as switching to a forced-choice format yields correct answers.", "body_md": "*Reading and editing the visual workspace of a vision-language model.*\n\nVision-language models hallucinate: ask one whether some object is in a picture and it will happily say *yes* whether it's there or not. Why is this?\n\nUsing [Anthropic's recent J-lens method](https://www.anthropic.com/research/global-workspace) I found that LLaVA-1.5-7B's internal state seems to register that the object is absent. The exact same evidence that the yes/no question ignores produces almost perfect answers when posed as a choice instead (\"is this a lamp or a dog?\").\n\nPerhaps then the hallucination can be shown to be a product of the question format and a failure to pull up things that the model already knows. The knowledge of the image turns out to be readable (and editable) inside the model using Anthropic's J-lens.\n\nScope: one model (LLaVA-1.5-7B), 13 categories, 39 core images, greedy decoding. Every number here comes out of cached data in the repo linked at the end.\n\nShow the model a photo with no lamp in it. Ask: *\"Is there a lamp in this image? Answer yes or no.\"*. Do it for 39 different lamp-less images and it says yes 39 times out of 39. (It also says yes 39/39 when the lamp really is there. There is a known yes-bias problem ([here](https://arxiv.org/pdf/2412.20622v1) and [here](https://arxiv.org/pdf/2605.04641)) with this family of models.)\n\nUsing the Jacobian lens, we can read the model's internal state during the hallucination. It seems like (if we rely only on the J-space) the model knows what is present and what is not.\n\nThe Jacobian lens is a way to turn a layer's raw activations into scores over the model's own vocabulary. In order to ask \"how readable is the word *lamp* right here?\", point it at the image tokens (the ~576 positions where the picture lives inside the prompt) and rank where the true word lands out of 32,000. Objects genuinely in the picture sit at a median rank of **16**. The suggested-but-absent lamp sits at **111,** roughly an order of magnitude weaker. An absent object that wasn't asked about sits at **430**. A plain logit lens shows the same separation (seen ~11 vs absent 282/3142). This is a fact about LLaVA, not about the Jacobian lens.\n\n[Neo et al. showed](https://arxiv.org/abs/2410.07149) object identity is decodable at these positions with a plain logit lens; the J-lens differs in that it transports the activation through the average Jacobian rather than unembedding it directly.\n\nThat jump from 430 down to 111 when the question named the word shows us that naming a word in the prompt makes it a little more readable at the image tokens regardless of the picture's content (plausibly the question word echoing in through attention). Even when the absent object is included in the prompt, it stays about an order of magnitude weaker in the workspace than anything the model really did see. There is some distinction here in the workspace between 'asked in the prompt' and 'actually seen in the image' that the LLM is not able to make when answering.\n\nWe are able to track when the model starts to lean towards answering \"yes\". At the position where the next word is chosen, you can measure how strongly the model leans Yes vs No at each layer. The lean is flat early, starts climbing around layer 13, peaks around layer 20, and settles positive at the output.\n\nThe curve for asking about a present and an absent object are almost the same. Since they agree on the verdict, maybe this is to be expected as the image stays the same and the prompt changes by only one word. The object being visible in the left graph (in the workspace) makes little difference to the result of the model here.\n\n*(Left: the internal workspace, present being separated from absent. Right: the answer being decided)*\n\nIs it just that the model *can't* use its visual evidence for this kind of question? I don't think so. Same 39 images and word pairs with a different question: *\"Which of these is in the image: a lamp or a dog?\"* Asked both ways round so it can't just pick the first option gives an accuracy of **38/39 and 37/39**.\n\nSo the evidence is present and readable the whole time. A question that names both options and forces a choice uses it well. A yes/no question about one option defaults to agreement. Note that the question was given before the image in these tests.\n\nAt least here, hallucination doesn't seem to be a perception failure, and isn't even about the evidence being overridden. The correct signal is inside the model linearly readable, but at natural evidence levels it's swamped out by the model's yes-bias. In cases of hallucination, the truly present objects can be read from the J-space instead of the output tokens.\n\nThe natural next step is to interfere with what the model sees in each image. Luckily we can edit the workspace. The lens gives you a direction for \"dog\" and a direction for \"cat\"; inside the model you can swap how much of each is present, at some dosage α (α=1 is an exact exchange). Editing the words output from the model gets us in a loop (\"cat cat cat...\"). Editing only the input positions (prompt, image tokens) works nicely. The natural ancestor to these tests is again [Neo et al.](https://arxiv.org/abs/2410.07149).\n\nIncreasing α predictably and smoothly changes the model's preference for what it saw to the imposter. There is a tie at α=1, and the full flip to the imposter happens past 1 (α ≈ 1.25–1.5). Swapping two unrelated words (as in, irrelevant to the real image) at the same strength doesn't move anything.\n\nSome image descriptions from the model before and after a word swap:\n\nThe Dalmatian swap: swap dog→cat and the broad category of animal flips, but the spots it saw survive the edit! Swapping cat into dogs of other breeds keeps attributes like the dogs smile or tongue.\n\nThe injected concept also drags in its own baggage. Swapping \"car\" with \"plane\" brings a sky with it that was not in the real image.\n\nWith two objects side by side, each has its own handle: swap the dog and the cup next to it is left untouched.\n\nLast one. The vision encoder here is a frozen CLIP, 24 layers of patch embeddings with no word vocab and no unembedding. Perhaps it can hold something word-shaped on its own, before the language model gets involved?\n\nI fit a J-lens for each encoder layer. The CLIP lens was averaged over 144 images and calibrated per word. I am hesitant to frame this as a genuine, category-general word channel, but from about encoder layer 10 up it seems something like a J-lens workspace is able to be found. Categories the lens had never seen an image of still read near the top.\n\nTwo things are needed before any signal shows up. First, read the patch-averaged activation, not individual patches. Second, calibrate per word. Calibration here is subtracting each candidate word's mean score across images because some words score high on every image and bury the signal (this is the same correction the original paper needed). With both, a seemingly category-general channel appears from about layer 10. The layer-22 lens reads layer-8 activations nearly as well as layer-8's own lens does, so this is one shared channel decoding a signal that's linearly present early, not something computed at layer 10.\n\nOf course a far weaker result than the Anthropic language-model side. This is picking the right word out of 45 after calibration compared to 32,000 in vocab for the LLM.\n\n[Gandelsman et al.](https://arxiv.org/abs/2310.05916) showed CLIP's internal contributions are text-interpretable, but via CLIP's *own* shared image-text embedding space, decomposing the direct effects on the final representation. This post reads intermediate CLIP layers through a Jacobian into a *different* model's vocabulary. So \"CLIP internals are word-readable\" is known, this specific readout mechanism is not.\n\nMost of this is known already. These models hallucinate, the phrasing of the question changes how often they do, as found in work like [POPE](https://arxiv.org/pdf/2305.10355). Of course so is the ability to decode more from a model's activations than it says out loud. The coordinate swap is from [the Anthropic paper](https://www.anthropic.com/research/global-workspace). The closest work I could find is [Neo et al.](https://arxiv.org/abs/2410.07149), the differences between this post and the paper is the cross-boundary framing and the three-way present/asked-absent/unasked-absent comparison.\n\nAs far as I can tell the Jacobian-lens method hadn't been pushed across a model's vision-language boundary before. I was happy to see information from the vision encoder susceptible to the same method and readable throughout the VLM.\n\nEverywhere that content is read at the language-model layers (the lamp readouts, the forced choice), a plain logit lens gives essentially the same answer, those results don't depend on the Jacobian lens. It only does something a logit lens can't when reading the frozen vision encoder internals. In CLIP a naive projection of an intermediate activation sits at chance, and only the measured Jacobian recovers the word. (The editing results use the lens directions rather than reading them off, so I haven't checked whether a logit lens would swap as cleanly.)\n\nThe repo: [https://github.com/jude-sph/J-lens-Vision](https://github.com/jude-sph/J-lens-Vision)\n\nCode and results were written with AI-assistance.", "url": "https://wpnews.pro/news/reading-into-vlm-hallucinations-using-the-jacobian-lens", "canonical_source": "https://www.lesswrong.com/posts/T3u6Hctes6vkawsib/reading-into-vlm-hallucinations-using-the-jacobian-lens", "published_at": "2026-07-10 00:51:21+00:00", "updated_at": "2026-07-10 01:13:13.156196+00:00", "lang": "en", "topics": ["artificial-intelligence", "large-language-models", "computer-vision", "ai-research", "ai-safety"], "entities": ["Anthropic", "LLaVA-1.5-7B", "Jacobian lens", "Neo et al."], "alternates": {"html": "https://wpnews.pro/news/reading-into-vlm-hallucinations-using-the-jacobian-lens", "markdown": "https://wpnews.pro/news/reading-into-vlm-hallucinations-using-the-jacobian-lens.md", "text": "https://wpnews.pro/news/reading-into-vlm-hallucinations-using-the-jacobian-lens.txt", "jsonld": "https://wpnews.pro/news/reading-into-vlm-hallucinations-using-the-jacobian-lens.jsonld"}}