When Retrieval Doesn't Help: A Large-Scale Study of Biomedical RAG

A large-scale study of biomedical question-answering systems found that retrieval-augmented generation (RAG) provides only small and inconsistent accuracy improvements—typically 1-2 percentage points—over models that do not use retrieval. Testing five open-weight models from 7B to 72B parameters across ten datasets, four retrieval methods, and four corpora, researchers determined that the choice of backbone model had a far greater impact on performance than the retrieval source or method. The findings indicate that the primary limitation is not retrieval quality but the models' inability to effectively use the retrieved evidence.

arXiv:2606.04127v1 Announce Type: new Abstract: Medical question answering is a high-stakes setting where factual errors can have serious consequences. Retrieval-augmented generation RAG is widely viewed as a promising solution, and prior work has reported substantial gains for large medical QA models. We revisit this assumption across a broad range of open-weight instruction-tuned models spanning 7B to 72B parameters. Across five models, ten biomedical QA datasets, four retrieval methods, and four retrieval corpora, we find that retrieval yields only small and inconsistent improvements over a no-retrieval baseline, typically within 1-2 points. In contrast, the choice of backbone model has a much larger effect than the choice of retriever or corpus, and expert and layman retrieval sources perform similarly in most settings. These results suggest that the main bottleneck is not retrieval quality alone, but the model's limited ability to use retrieved evidence effectively.