When Search Agents Should Ask: DiscoBench for Clarification-Aware Deep Search

Researchers introduced DiscoBench, a benchmark for clarification-aware deep search, to evaluate whether LLM-powered search agents can identify ambiguity, ask effective questions, and recover correct reasoning paths. The benchmark contains 211 samples across 11 domains, and experiments show that current agents often fail to ask for clarification, performing worse than direct guessing.

arXiv:2606.27669v1 Announce Type: new Abstract: Search agents powered by large language models LLMs are increasingly used to solve complex information-seeking tasks, requiring multi-step retrieval and reasoning to fulfill user goals. However, existing benchmarks often assume that user queries are complete and explicit, overlooking the fact that real-world search requests are frequently vague, underspecified, or even factually incorrect. In deep search scenarios, such ambiguity can propagate along multi-step reasoning chains and lead agents toward incorrect search trajectories. To address this gap, we introduce DiscoBench, a benchmark for clarification-aware deep search, designed to evaluate whether search agents can proactively identify ambiguity, ask effective clarification questions, and recover correct reasoning paths through user interaction. DiscoBench contains 211 samples and 463 ambiguity instances across 11 real-world domains, covering four ambiguity types. We further design a user simulator for multi-turn interaction and evaluate model performance from four perspectives: task utility, ambiguity detection, interaction strategy, and cost efficiency. Experiments on representative LLMs show that ambiguity detection and effective clarification are distinct capabilities, and that repeatedly searching instead of asking for clarification often performs worse than direct guessing, highlighting a critical gap between retrieval ability and interactive problem-solving in current search agents.