Detect Before You Disambiguate: Perturbation-Based Input Ambiguity Detection in LLMs

A new arXiv paper (2610.36353v1) proposes a label-preserving perturbation framework for detecting ambiguous inputs to large language models before the models attempt to answer them. The authors instantiate the framework with two detectors — a black-box rule-based perturbation library and a white-box importance-guided method using model-internal signals — and report that across four datasets spanning open-domain, conversational and visual QA, both outperform output-side uncertainty baselines and LLM-paraphrase-based perturbation baselines. The work argues that ambiguity must be detected before disambiguation, since LLM-generated clarifications can resolve the ambiguity before it is measured, and the code is released on GitHub.

arXiv:2610.36353v1 Announce Type: new Abstract: Large language models LLMs often answer ambiguous inputs as if they had a single intended meaning. This is problematic because input ambiguity is an upstream source of uncertainty: when a prompt admits multiple valid interpretations, the key question is whether the model should answer directly at all. Existing input uncertainty methods often rely on LLM-generated statements or clarifications, but such transformations may already disambiguate the input before the ambiguity is measured. We propose a label-preserving perturbation framework for input ambiguity detection in which perturbations remain close to the original input while preserving its label. We instantiate this principle with two detectors: a black-box rule-based perturbation library and a white-box importance-guided method using model-internal signals. Experiments across four datasets in open-domain, conversational, and visual QA show that our methods outperform output-side uncertainty baselines and LLM-paraphrase-based perturbation baselines. Our results highlight the importance of detecting ambiguity before attempting to disambiguate it. Our code is available \href{https://github.com/Xiao0o0o/input-ambiguity-detection}{\textcolor{purple}{here}}.