Staying VIGILant: Mitigating Visual Laziness via Counterfactual Visual Alignment in MLLMs

Researchers propose VIGIL, a reinforcement-learning post-training framework that reduces hallucinations in multimodal large language models by maximizing mutual information between visual input and generated responses. The method outperforms existing alignment techniques on hallucination and reasoning benchmarks, achieving state-of-the-art results with only 25% of preference data and demonstrating emergent spatial grounding without explicit supervision.

arXiv:2606.26387v1 Announce Type: new Abstract: Multimodal large language models MLLMs extend large language models LLMs with visual perception, enabling joint reasoning over images and text. Despite inheriting strong reasoning capabilities from LLMs, they remain prone to hallucinations that contradict their visual inputs. Mechanistic studies indicate that this weakness stems from visual laziness: MLLMs encode the correct visual evidence internally, but overly rely on strong language priors during response. Existing alignment methods, such as direct preference optimization, primarily optimize outcome-level rewards based on text. This introduces an optimization bias toward linguistic shortcuts, leading to responses that often contradict the visual evidence. To address this, we propose Visual Information Gain In aLignment VIGIL , a reinforcement-learning RL post-training framework that shifts the focus from numerical reward fitting to causal visual grounding. VIGIL introduces a geometric constraint that explicitly maximizes the mutual information between the visual input and the generated response. We achieve this by penalizing "blind confidence" instances where the model remains improperly certain even when textual-visual attention is masked to create a counterfactual blind state. Extensive experiments show that VIGIL consistently outperforms recent alignment methods across hallucination and reasoning benchmarks without compromising text-only capabilities. Our approach matches the full-data performance of state-of-the-art methods using only 25% of the preference data and even demonstrates emergent spatial grounding capabilities without explicit bounding box supervision.