TRIAD: Benchmarking Omni-Modal Ambiguity in Multimodal Large Language Models

A new benchmark, TRIAD, finds that the best of 14 omni-modal large language models trails humans by more than 20 points in scene-level group accuracy on ambiguity resolution, even though item-level scores look strong. TRIAD, described in arXiv paper 2610.30841v1, contains 327 hand-written text-image-audio queries in 86 scene groups spanning 18 ambiguity categories, each built so the full triplet yields a unique answer while dropping any one modality leaves multiple consistent options. Removing answer options pushes model group accuracy into the single digits while humans stay robust, and leave-one-out and transcript/description counterfactuals show current systems often underuse audio cues when text and image are present.

arXiv:2610.30841v1 Announce Type: new Abstract: Ambiguity is a central feature of natural communication: a sentence, image, or sound may remain underspecified until evidence from the other channels is considered. Yet current multimodal benchmarks mostly test perception, recognition, or reasoning over already determinate inputs, leaving unclear whether omni-modal large language models can resolve ambiguity distributed across text, vision, and audio. We introduce TRIAD, a benchmark for omni-modal ambiguity resolution. Each item is a text--image--audio triplet paired with a question and a set of answer options, and is constructed so that the full triplet determines a unique answer while removing any one modality makes the set of consistent options non-unique. TRIAD contains $327$ hand-written queries in $86$ scene groups and covers $18$ ambiguity categories across the three modalities. Evaluating $14$ omni-modal systems reveals that strong item-level performance does not translate into robust scene-level disambiguation: the best model still trails humans by over $20$ points in group accuracy. Removing answer options widens the gap further, driving model group accuracy to the single digits while humans remain robust. Leave-one-out and transcript/description counterfactuals further show that current systems often underuse audio cues when text and image are present. TRIAD reframes omni-modal evaluation around disambiguation rather than recognition, and provides a taxonomy, benchmark, and diagnostic protocol for measuring this capability.