The Bottleneck Is Retrieval, Not Reading: Diagnosing Evidence-Based Generation with RAG for Long Context Question Answering

A study posted to arXiv (2610.26951v1) introduces evidence-based generation (EBG), a method requiring models to return answers with line-level citations, plus a joint Ans+Evid metric that scores correctness and grounding together. Testing ten proprietary frontier and three open-weight LLMs on reformulated benchmarks — MultiHop-RAG, QASPER-MP, a synthetic two-hop haystack, and an evidence-augmented NoLiMa spanning 10K to 1M characters — the authors found retrieval, not reading, is the bottleneck: similarity retrieval surfaces any relevant chunk far more reliably than all required chunks for multi-fact queries, and chunking destroys document meta-information needed to resolve identity-bearing queries. EBG also exposed grounding failures invisible to answer-only evaluation, including incomplete citations in two-needle settings and a widening Ans-versus-Ans+Evid gap when context contradicts parametric knowledge; data and code are at github.com/debobanerjee/LC-EBG.

arXiv:2610.26951v1 Announce Type: new Abstract: Retrieval-augmented generation RAG is the dominant paradigm for grounding large language models in external corpora, but its similarity-driven, chunked pipeline has structural blind spots: it cannot anticipate which passages a question requires and discards document-level context that resolves cross-passage references. We introduce evidence-based generation EBG , requiring models to return an answer with line-level citations, and a joint Ans+Evid metric that scores correctness and grounding simultaneously. We reformulate two corpora into evidence-grounded multi-document QA benchmarks- MultiHop-RAG and QASPER-MP, a multi-paper adaptation of QASPER with realistic short paper cues, and pair them with a controlled synthetic two-hop haystack and an evidence-augmented NoLiMa spanning 10K-1M characters. Across ten proprietary frontier and three open-weight LLMs, EBG isolates two RAG limitations that long-context models avoid: similarity retrieval surfaces any relevant chunk far more reliably than all required chunks for multi-fact queries, and chunking destroys document meta-information needed to resolve identity-bearing queries. EBG also exposes grounding failures invisible to answer-only evaluation, including incomplete citations in two-needle settings and a widening gap between Ans and Ans+Evid when context contradicts parametric knowledge. Data and code are available at https://github.com/debobanerjee/LC-EBG.