Mitigating Hallucinations in Large Vision-Language Models via Causal Route Gating

Researchers have identified that hallucinations in large vision-language models (LVLMs) stem from route competition, where textual pathways override visual evidence during token decision-making. To address this, the team introduced a training-free intervention that decomposes attention heads into visual and text routes, selectively suppressing only the text route while preserving visual processing. The method reduced hallucination-related errors across five benchmarks with minimal impact on overall multimodal performance and modest inference overhead.

arXiv:2605.24024v1 Announce Type: new Abstract: Large vision-language models LVLMs often hallucinate content that is fluent yet unsupported by the image, limiting their reliability in real-world deployment. We show that a key failure mode arises from route competition: even when visual tokens receive attention, the final token decision can be dominated by the textual pathway, causing the decoder to follow linguistic priors over visual evidence. To mitigate this, we propose a training-free, decision-aligned intervention that decomposes each attention head into a visual route and a text route, and estimates their token-level effects using an efficient one-forward/one-gradient approximation. These estimates reveal route conflict within heads and identify prior-dominant ones, enabling selective suppression of only the text route while keeping the visual route intact. Across five benchmarks spanning discriminative and generative settings, our method consistently reduces hallucination-related errors across models with limited impact on overall multimodal performance, while incurring a modest inference-time overhead.