arXiv:2611.17954v1 Announce Type: new Abstract: Human Label Variation (HLV) is increasingly recognised as an informative signal that reflects meaningful differences in how people interpret the same input. While HLV has been studied extensively in Natural Language Inference (NLI), much less is known about HLV in Visual Entailment (VE; Xie et al., 2019), where the textual premise is replaced by an image, keeping the hypothesis and label space fixed. We introduce VariNLI-VE, a dataset for studying how HLV changes between paired NLI and VE instances. We re-annotate a subset of originally neutral SNLI (Bowman et al., 2015) instances and their visual counterparts, with the same annotators evaluating both the textual and visual versions. Each annotation includes a label, confidence score, and explanation, yielding 2,250 annotations. Labels and explanations are collected jointly, providing ecologically valid explanations. Because the image may reveal information not stated in the text, one might expect disagreement to decrease in VE. Instead, we find no clear overall reduction: disagreement decreases for some instances while it increases for others. Explanations reveal further variation, as annotators can retain the same label while changing their rationale. When evaluating vision-language models (VLMs) on the paired data, we find lower accuracy, weaker alignment with human label distributions, and generally higher uncertainty in VE than in NLI.
In the Eye of the Beholder: Human Label Variation in Textual and Visual Entailment
A new arXiv paper (2611.17954v1) introduces VariNLI-VE, a dataset of 2,250 annotations built by re-annotating a subset of originally neutral SNLI instances and their visual counterparts with the same annotators, to study how Human Label Variation differs between Natural Language Inference and Visual Entailment. The authors report no clear overall reduction in disagreement in Visual Entailment — disagreement falls for some instances and rises for others — and find that vision-language models show lower accuracy, weaker alignment with human label distributions, and generally higher uncertainty on Visual Entailment than on NLI.
Run your AI side-project on zahid.host
EU-based hosting, git-push deploys, automatic HTTPS, no cold starts. Free tier with a custom domain — perfect for shipping the agent you just read about.