Simple BM25 outperforms agents on large corpora

A new study finds that lexical BM25 retrieval outperforms sophisticated agentic search methods on large corpora, achieving 50.5 accuracy versus 30.7 for File-System Agent at the largest scale. The gap widens with corpus size, and BM25 uses 39 times fewer query tokens, suggesting it should be the default for large-scale retrieval.

Lexical BM25 now eclipses sophisticated agentic search once a collection grows beyond a modest size, overturning the assumption that learned explorers are the default for massive retrieval tasks. The surprise lies in how a decades‑old formula reclaims the top spot without any LLM‑driven construction. Until recently, evaluation of Retrieval‑Augmented Generation pipelines mixed disparate benchmarks and fixed corpus scales, reporting dense retrievers, graph‑based indexes, and sequential agents such as File‑System Agent as competitive alternatives to plain BM25. Those studies never exposed how performance evolves when the same documents are scaled by orders of magnitude. At the largest corpus tier, BM25 achieves an accuracy of 50.5 , outpacing the File‑System Agent’s 30.7 and DenseRAG’s 29.9 points 1 https://arxiv.org/abs/2607.26497 . The gap widens as more tokens are added, reaching a margin of nearly twenty points at full scale. File‑System Agent’s sequential exploration consumes 39 times more query tokens than a single BM25 pass on the identical bedrock, and its effectiveness deteriorates as the search space expands 1 https://arxiv.org/abs/2607.26497 . The token overhead alone makes the agent prohibitively expensive for large‑scale deployments. Around 10 million corpus tokens, BM25 overtakes the File‑System Agent and becomes the dominant method across every larger shared tier 1 https://arxiv.org/abs/2607.26497 . Beyond this crossover point, global candidate ranking consistently outperforms localized, stepwise discovery. The study isolates a single reader model and judging protocol, so the reported superiority may not generalise to alternative downstream heads or evaluation metrics; moreover, the authors note that “agentic reasoning works best after ranked discovery rather than in place of it.” This suggests a hybrid pipeline could still benefit from lightweight agents once BM25 has narrowed the candidate set. Practitioners should adopt BM25 as the default retrieval layer for any corpus exceeding a few million tokens and reserve agentic components for post‑ranking refinement, thereby cutting engineering complexity and token cost while preserving—or improving—overall accuracy.