Looks the Same, Answers Differently: Flip-Direction Steering for Robust Vision-Language Reasoning

Researchers introduced FlipDir (Flip-Direction Steering), a training-free inference-time method that estimates a low-rank flip-inducing activation subspace from contrastive pairs of original and answer-flipping inputs and selectively steers hidden states during decoding to stabilize vision-language model reasoning. The work, posted as arXiv:2609.28851v1, also presents VisFlip, a benchmark spanning nine dataset-variation combinations across scientific reasoning, robot-scene understanding, and medical VQA, and reports that FlipDir outperforms existing methods on a combined recovery and preservation metric across 18 settings. The authors state the code will be made publicly available.

arXiv:2609.28851v1 Announce Type: new Abstract: Vision-language models VLMs achieve strong visual reasoning performance, yet subtle changes from routine image capture and processing can alter their reasoning trajectories even when images appear nearly identical. In long-horizon generation, the resulting activation shifts may accumulate across decoding steps, progressively altering reasoning tokens and ultimately changing the final answer, a phenomenon referred to as answer flips. To address this instability, we propose FlipDir Flip-Direction Steering , a training-free inference-time method that estimates a low-rank flip-inducing activation subspace from contrastive pairs of original and answer-flipping inputs and selectively steers hidden states during decoding. A margin-based gate limits subspace attenuation to uncertain decoding steps, recovering original predictions while preserving stable ones. To evaluate robustness beyond accuracy or consistency on fixed test sets, we introduce VisFlip, a benchmark framework that constructs evaluation groups for a target model and visual variation setting to separately assess recovery of original predictions and preservation of stable ones. VisFlip spans nine dataset-variation combinations across scientific reasoning, robot-scene understanding, and medical VQA, covering subtle visual variations common in each domain. Experiments across 18 settings demonstrate that FlipDir consistently outperforms existing methods on the combined recovery and preservation metric. We will make our code publicly available.