AmbiSteer: Selectively Eliciting Ambiguity-Aware Responses in Vision-Language Models A lightweight framework called AmbiSteer raised the share of vision-language model inputs handled appropriately from 59.0% to 77.0% on average across three VLMs by using a linear classifier to predict whether a question is ambiguous and conditioning activation steering on that prediction, according to the arXiv paper 2611.17900v1. The authors report that steering alone had limited and model-dependent effects while prompting was generally more effective, and that combining prompting and steering substantially increased clarification and enumeration behaviors but also triggered them on unambiguous cases. AmbiSteer's classifier-gated steering is presented as a step toward reducing model overconfidence in uncertain multimodal settings such as referentially ambiguous Visual Question Answering. arXiv:2611.17900v1 Announce Type: cross Abstract: Vision-language models VLMs often exhibit overconfidence, providing firm answers even when visual evidence is insufficient. A prominent case is referential ambiguity in Visual Question Answering VQA , where multiple entities in an image are plausible referents of a question. Addressing such cases requires recognizing when ambiguity is present: ambiguous questions can be handled by asking for clarification or enumerating plausible referents, while unambiguous questions should receive a direct answer. Current VLMs struggle with this distinction, often defaulting to a single interpretation. To investigate whether activation steering can serve as an inference-time mechanism to elicit such behaviors, we introduce AmbiSteer, a lightweight framework to selectively elicit ambiguity-aware behavior. Across three VLMs, we find that steering alone has limited and model-dependent effects, while prompting is generally more effective. Combining prompting and steering substantially increases clarification and enumeration behaviors, but also elicits them on unambiguous cases. AmbiSteer addresses this using a linear classifier to predict input ambiguity and condition the steering direction accordingly, increasing the proportion of inputs handled appropriately from 59.0% to 77.0% on average across models. Our work sets the ground for lightweight approaches to reduce model overconfidence in uncertain multimodal settings.