content type paperpublished August 2026 DeepAmbigQA: Ambiguous Multi-hop Questions for Benchmarking LLM Answer Completeness
AuthorsJiabao Ji†**, Min Li, Priyanshu Kumar, Shiyu Chang†, Saloni Potdar
DeepAmbigQA: Ambiguous Multi-hop Questions for Benchmarking LLM Answer Completeness
AuthorsJiabao Ji†**, Min Li, Priyanshu Kumar, Shiyu Chang†, Saloni Potdar
Large language models (LLMs) with integrated search tools show strong promise in open-domain question answering (QA), yet they often struggle to produce complete answer set to complex questions such as “Which actor from the film Heat won at least one Academy Award?”, which requires (1) distinguishing between multiple films sharing the same title and (2) reasoning across a large set of actors to gather and integrate evidence. Existing QA benchmarks rarely evaluate both challenges jointly. To address this, we introduce DEEPAMBIGQAGEN, an automatic data generation pipeline that constructs QA tasks grounded in text corpora and linked knowledge graph, generating natural and verifiable questions that systematically embed name ambiguity and multi-step reasoning. Based on this, we build DEEPAMBIGQA, a dataset of 3,600 questions requiring multi-hop reasoning and half of them explicit name ambiguity resolving. Experiments reveal that, even state-of-the-art GPT-5 show incomplete answers, achieving only 0.13 exact match on ambiguous questions and 0.21 on non-ambiguous questions. These findings highlight the need for more robust QA systems aimed at information gathering and answer completeness.
Can Open Domain Question Answering Models Answer Visual Knowledge Questions?
February 28, 2022research area Speech and Natural Language Processing
The task of Outside Knowledge Visual Question Answering (OKVQA) requires an automatic system to answer natural language questions about pictures and images using external knowledge. We observe that many visual questions, which contain deictic referential phrases referring to entities in the image, can be rewritten as “non-grounded” questions and can be answered by existing text-based question answering systems. This allows for the reuse of…
Question Rewriting for End to End Conversational Question Answering
February 21, 2021research area Knowledge Bases and Search, research area Speech and Natural Language Processingconference WSDM
Conversational question answering (QA) requires the ability to correctly interpret a question in the context of previous conversation turns. We address the conversational QA task by decomposing it into question rewriting and question answering subtasks. The question rewriting (QR) subtask is specifically designed to reformulate ambiguous questions, which depend on the conversational context, into unambiguous questions that can be correctly…