Transcoders Trace Visual Grounding and Hallucinations in Vision-Language Models

Researchers have introduced Transcoders, a function-centric framework that decomposes vision-language models into interpretable computational pathways linking image patches to token generation. Applied to Gemma 3-4B-IT, the method produces stronger and more stable effects on visually grounded tokens than traditional Sparse Autoencoders, and enables a logistic classifier to predict hallucinations at AUC 0.68 using graph-based circuit features. The findings demonstrate that function-centric circuit decomposition provides interpretable and predictive accounts of multimodal computation in vision-language models.

arXiv:2605.22902v1 Announce Type: new Abstract: Generative Vision-Language Models VLMs perform well on multimodal reasoning, but how visual inputs are transformed to text remains poorly understood. Existing interpretability work on VLMs uses Sparse Autoencoders SAEs , which decompose static residual representations and miss the functional updates that drive cross-modal interaction. We adopt a function-centric framework based on Transcoders, sparse approximations of MLP sublayers that act as a causal proxy for layer-wise computation. Applied to Gemma 3-4B-IT, the framework decomposes the model into interpretable computational pathways linking image patches to directions in token generation. Transcoder attributions produce stronger and more stable effects on visually grounded tokens under patch ablation than SAE attributions, and align better with semantically relevant image regions. A False Visual Grounding counterfactual analysis confirms that the recovered pathways are specific to vision-language interaction.Finally, we perform a structural analysis of hallucinated generations, by extracting graph-based indicators from circuit traces produced by the transcoders. A logistic classifier over these mechanistic graph features predicts hallucinations at AUC $0.68$. These results show that function-centric circuit decomposition yields interpretable and predictive accounts of multimodal computation in VLMs.