Evidence-Aware RAG Evaluation: Why Ground Truth Coverage Determines Retrieval Scores in Clinical Agent Systems

A clinical AI research team published a paper demonstrating that RAG retrieval scores are shaped by ground-truth annotation coverage, not retriever quality alone, using a breast lesion management system as a testbed. A second curation pass on an 18-document, 18-query corpus grew evidence spans from 25 to 37 and chunk targets from 36 to 51 — a 41.7% larger denominator — while Recall@3 stayed at 0.94 and retrieval rankings were unchanged. The team proposes an evidence-aware evaluation protocol that treats annotation coverage as a first-class variable and separates retrieval failures from generation failures, with a three-stage BM25-plus-dense, cross-encoder reranked pipeline at 384-token chunks and 20% overlap retrieving accepted evidence in the top three for 17 of 18 queries.

Your RAG system's retrieval scores depend on how much of your corpus you've annotated, not on retriever quality alone. A new paper from a clinical AI team demonstrates this measurement artifact in a breast lesion management system and proposes an evidence-aware evaluation protocol that separates annotation coverage from system quality. The problem surfaces when you build evals for high-stakes agent systems. You can't exhaustively label every relevant passage in a large corpus. Your ground truth becomes a sample, and your retrieval metrics reflect that sample's completeness as much as your system's performance. Standard RAG evaluation assumes your ground truth annotations are complete. You label relevant chunks for each query, run your retriever, and measure precision and recall. But in practice: The clinical team tested this with an 18-document corpus and 18 queries about breast lesion management. Initial curation produced 25 evidence spans mapped to 36 chunk targets. After a second pass, they found 37 spans and 51 chunks without changing retrieval rankings. Recall@3 stayed at 0.94, but the denominator grew by 41.7%. The system retrieved the same chunks, but the eval dataset revealed more relevant targets. In any RAG system over a corpus larger than a few dozen documents, incomplete annotation is the norm. The paper quantifies this: a 48% increase in evidence spans 25 to 37 and a 42% increase in chunk targets 36 to 51 occurred between curation rounds, with zero change to retrieval output. The protocol treats annotation coverage as a first-class variable. Instead of pretending your ground truth is exhaustive, you track: For overlapping chunks, the protocol considers all chunks with equivalent evidence as relevant. If your chunking strategy creates three 384-token chunks that each contain the same critical sentence, all three count as correct retrievals. The system uses a three-stage retriever: Pseudocode illustrating three-stage retrieval pipeline Stage 1: Parallel sparse and dense retrieval bm25 results = bm25 index.search query, top k=100 dense results = embedding model.search query, top k=100 Stage 2: Merge and deduplicate merge results is a custom function that unions result sets and removes duplicate chunk IDs candidate pool = merge results bm25 results, dense results Stage 3: Cross-encoder reranking reranked = cross encoder.rerank query=query, candidates=candidate pool, top k=5 The cross-encoder reranker consistently outperformed single-stage retrievers. With 384-token chunks and 20% overlap, it retrieved accepted evidence in the top three results for 17 of 18 queries. The team tested chunk sizes from 128 to 1024 tokens with overlaps from 0% to 20%. Results from the paper's evaluation: | Chunk Size | Overlap | Recall@3 | R-Precision | Notes | |---|---|---|---|---| | 128 tokens | 0% | 0.78 | 0.72 | Fragments evidence spans | | 384 tokens | 20% | 0.94 | 0.94 | Selected configuration | | 512 tokens | 0% | 0.89 | 0.83 | Misses boundary cases | | 1024 tokens | 20% | 0.83 | 0.78 | Dilutes relevance signal | The 384-token chunk with 20% overlap balances two failure modes. Smaller chunks fragment evidence and hurt recall. Larger chunks dilute the relevance signal and hurt precision. Overlap catches evidence that spans chunk boundaries. Retrieval is only half the system. The generation layer was evaluated separately using: The separation matters for observability. When your agent produces a bad answer, you need to know whether retrieval failed wrong chunks in context or generation failed right chunks, wrong synthesis . The evidence-aware protocol requires tracking annotation state across curation rounds. The team's workflow: This creates a feedback loop between system performance and ground truth quality. Your retrieval system becomes a discovery tool for improving your eval dataset. The protocol exposes three distinct failure modes: Retrieval failure : System doesn't surface relevant chunks. Fix with better embeddings, reranking, or chunking strategy. Annotation gap : System retrieves relevant chunks that aren't labeled. Fix with more curation rounds. Evidence insufficiency : Corpus doesn't contain the answer. Fix with corpus expansion or query refinement. Standard RAG evals conflate these. A low recall score could mean any of the three. Evidence-aware evaluation separates them. For production systems: Clinical RAG systems operate under HIPAA constraints. The evaluation protocol doesn't change compliance requirements, but it does affect audit trails: If you're using an external LLM for evaluation, you're sending query-chunk pairs outside your perimeter. Consider local models or anonymization pipelines. Use evidence-aware evaluation when: Avoid it when: The protocol adds overhead. You're tracking annotation rounds, coverage estimates, and evidence provenance. But for clinical agents and other high-stakes systems, the alternative is shipping evals that lie about system quality because they conflate retrieval performance with annotation gaps.