Visuals Lie, Consistency Speaks: Disentangling Spatial Attention from Reliability in Vision-Language Models

Researchers at arXiv challenge the common assumption that visual attention correlates with reliability in vision-language models. Their VLM Reliability Probe study across multiple models finds that spatial attention metrics have near-zero correlation with accuracy, while self-consistency in generation dynamics is the dominant predictor of truth. The work reveals architectural divergences in how models encode reliability, with implications for building more trustworthy multimodal AI systems.

arXiv:2606.17389v1 Announce Type: new Abstract: Multimodal Foundation Models are increasingly used as reasoning agents, making reliability, knowing when a model may hallucinate, critical. A common intuition, which we call the Attention-Confidence Assumption, holds that reliability follows from "structural" visual perception: tight attention on relevant regions should signal a trustworthy answer, while scattered attention signals confusion. We challenge this through the VLM Reliability Probe VRP , a systematic cross-family study of reliability signals in contemporary Vision-Language Models VLMs . We introduce structural-attention metrics, cluster counts C k and spatial entropy H s , to quantify the visual encoder's gaze, and track its evolution Delta H s across layers. This reveals a "Symbolic Detachment": models often "Early Lock" visual features only to diffuse attention later, severing early perception from final generation. Contrary to the grounding hypothesis, we find a "Cluster Failure": spatial attention has near-zero correlation R approx 0.001 with accuracy. Instead, reliability is a phenomenon of generation dynamics and internal-state distributions. Self-Consistency, the agreement rate across sampled reasoning paths, is the dominant predictor of truth R = 0.429 . Scaling causal interventions exposes a sharp architectural divergence: LLaVA locks its prediction in a fragile late-stage bottleneck, whereas PaliGemma and Qwen2-VL distribute reliability globally, staying resilient even when ~50% or more of their most predictive layer is destroyed. For current VLMs, reliability signals are detached from visual grounding maps and are best inferred from generation-time dynamics and hidden-state probes.