LaViSA: A Language and Vision Structural Ambiguity Benchmark

Researchers introduced LaViSA, a benchmark to evaluate vision-language models' ability to resolve structural ambiguity using visual scenes. Tests on proprietary and open-source models showed they can partially leverage visual cues but struggle with certain ambiguity types and subtle semantic distinctions.

arXiv:2606.19552v1 Announce Type: new Abstract: Structural ambiguity arises when a single sentence admits multiple valid interpretations due to its syntactic structure, posing a fundamental challenge for language understanding. Visual scenes serve as useful cues for resolving such ambiguity, and Vision and Language Models VLMs need to be capable of deriving possible semantic interpretations from visual scenes. We introduce Language and Vision Structural Ambiguity LaViSA , a benchmark designed to evaluate the ability of VLMs to resolve structural ambiguity leveraging visual scenes. LaViSA consists of ambiguous sentences, their disambiguated sentences, and corresponding images of these disambiguated sentences across seven ambiguity categories. Using LaViSA, we conduct a comprehensive evaluation of diverse VLMs, including both proprietary and open-source models with varying parameter scales and reasoning capabilities. Experimental results show that although recent VLMs can leverage visual scenes to resolve structural ambiguity to a some extent, they still struggle with certain ambiguity types and visually subtle semantic distinctions, indicating remaining limitations in resolving structural ambiguity using visual scenes.