Scan or Index? large Evidence Coverage, Answer Quality, and Cost in Long-Document RAG

A study of 11 retrieval configurations on 396 QASPER questions and 80 questions from eight complete textbooks found that Hybrid + Jev achieved the highest observed evidence coverage on books at 94.1%, versus 88.7% for Jev full scan and 81.3% for OpenAI dense retrieval. Higher evidence coverage did not consistently yield better answers: Hybrid + Decisions posted the highest observed book Answer Accuracy among retrieval methods at 0.892 despite recovering less reference evidence, and full scanning incurred substantially higher latency on books while reusable dense retrieval had lower projected API cost than paid decision-based methods as queries accumulate. The authors conclude that evidence retention, answer quality and reuse cost are complementary criteria for choosing a long-document retrieval strategy, noting most QASPER articles already fit within the hybrid candidate pool, which limits the benefit of scanning.

arXiv:2610.28957v1 Announce Type: new Abstract: Scanning every passage in a long document seems thorough, but does it preserve more useful evidence or produce better answers? We compare 11 retrieval configurations spanning lexical and dense search, hybrid reranking, and full-document scanning with Jev, OpenAI Decisions, and Cloudflare Clef. Our evaluation uses 396 QASPER questions and 80 questions from eight complete textbooks, with common passages, a five-passage limit, and a fixed answer generator. On books, Hybrid + Jev achieves the highest observed evidence coverage 94.1% , compared with 88.7% for Jev full scan and 81.3\% for OpenAI dense retrieval. Full scans examine every passage initially, yet later reduction steps can discard relevant evidence. Higher evidence coverage also does not consistently produce better answers: Hybrid + Decisions achieves the highest observed book Answer Accuracy among retrieval methods 0.892 , despite recovering less reference evidence. Full scanning incurs substantially higher latency on books under the tested deployments, while reusable dense retrieval has lower projected API cost than paid decision-based methods as queries accumulate. Most QASPER articles already fit within the hybrid candidate pool, limiting the benefit of scanning. These results highlight evidence retention, answer quality, and reuse cost as complementary criteria for choosing a long-document retrieval strategy.