A Plausible Explanation Is Enough to Mislead Multimodal Judges

A new arXiv paper (2610.22631v1) reports that plausible-sounding explanations mislead multimodal large language model judges, reducing their rejection of incorrect answers by 12.7 percentage points for plain justifications and 13.8 percentage points for chain-of-thought rationales across 14 MLLM judges. The authors introduce JUPE, a visual-question-answering benchmark that holds answers fixed while varying explanations, and name the failure "plausibility capture." Using signal detection theory, they find a significant mean shift in decision criterion toward acceptance without a clear change in mean discriminability, and note that tested mitigations offer limited, model-dependent protection — a risk that can inflate performance estimates when judges evaluate answers with explanations.

arXiv:2610.22631v1 Announce Type: new Abstract: Multimodal large language models MLLMs increasingly serve as judges in training and evaluation pipelines, often assessing answers together with explanations. Although AI judges are sensitive to textual factors, it remains unclear whether ordinary plausible explanations can sway judgments requiring visual verification. We introduce JUPE, a visual-question-answering VQA -style benchmark that fixes answers while varying explanations. Across 14 MLLM judges, ordinary explanations that sound plausible mislead judges, with effects comparable on average to deliberately persuasive arguments. Plain justifications and chain-of-thought rationales reduce rejection of incorrect answers by 12.7 and 13.8 percentage points, respectively. We call this failure plausibility capture. Using signal detection theory, we find a significant mean shift in decision criterion toward acceptance, without a clear change in mean discriminability. Explanations sway judgments across domains, explanation generators, and judging protocols, while tested mitigations offer limited, model-dependent protection. Such shifts can inflate performance estimates when judges evaluate answers with explanations.