Your RAG system's retrieval scores depend on how much of your corpus you've annotated, not on retriever quality alone. A new paper from a clinical AI team demonstrates this measurement artifact in a breast lesion management system and proposes an evidence-aware evaluation protocol that separates annotation coverage from system quality.
The problem surfaces when you build evals for high-stakes agent systems. You can't exhaustively label every relevant passage in a large corpus. Your ground truth becomes a sample, and your retrieval metrics reflect that sample's completeness as much as your system's performance.
Standard RAG evaluation assumes your ground truth annotations are complete. You label relevant chunks for each query, run your retriever, and measure precision and recall. But in practice:
The clinical team tested this with an 18-document corpus and 18 queries about breast lesion management. Initial curation produced 25 evidence spans mapped to 36 chunk targets. After a second pass, they found 37 spans and 51 chunks without changing retrieval rankings. Recall@3 stayed at 0.94, but the denominator grew by 41.7%. The system retrieved the same chunks, but the eval dataset revealed more relevant targets.
In any RAG system over a corpus larger than a few dozen documents, incomplete annotation is the norm. The paper quantifies this: a 48% increase in evidence spans (25 to 37) and a 42% increase in chunk targets (36 to 51) occurred between curation rounds, with zero change to retrieval output.
The protocol treats annotation coverage as a first-class variable. Instead of pretending your ground truth is exhaustive, you track:
For overlapping chunks, the protocol considers all chunks with equivalent evidence as relevant. If your chunking strategy creates three 384-token chunks that each contain the same critical sentence, all three count as correct retrievals.
The system uses a three-stage retriever:
bm25_results = bm25_index.search(query, top_k=100)
dense_results = embedding_model.search(query, top_k=100)
candidate_pool = merge_results(bm25_results, dense_results)
reranked = cross_encoder.rerank(
query=query,
candidates=candidate_pool,
top_k=5
)
The cross-encoder reranker consistently outperformed single-stage retrievers. With 384-token chunks and 20% overlap, it retrieved accepted evidence in the top three results for 17 of 18 queries.
The team tested chunk sizes from 128 to 1024 tokens with overlaps from 0% to 20%. Results from the paper's evaluation:
| Chunk Size | Overlap | Recall@3 | R-Precision | Notes |
|---|---|---|---|---|
| 128 tokens | 0% | 0.78 | 0.72 | Fragments evidence spans |
| 384 tokens | 20% | 0.94 | 0.94 | Selected configuration |
| 512 tokens | 0% | 0.89 | 0.83 | Misses boundary cases |
| 1024 tokens | 20% | 0.83 | 0.78 | Dilutes relevance signal |
The 384-token chunk with 20% overlap balances two failure modes. Smaller chunks fragment evidence and hurt recall. Larger chunks dilute the relevance signal and hurt precision. Overlap catches evidence that spans chunk boundaries.
Retrieval is only half the system. The generation layer was evaluated separately using:
The separation matters for observability. When your agent produces a bad answer, you need to know whether retrieval failed (wrong chunks in context) or generation failed (right chunks, wrong synthesis).
The evidence-aware protocol requires tracking annotation state across curation rounds. The team's workflow:
This creates a feedback loop between system performance and ground truth quality. Your retrieval system becomes a discovery tool for improving your eval dataset.
The protocol exposes three distinct failure modes:
Retrieval failure: System doesn't surface relevant chunks. Fix with better embeddings, reranking, or chunking strategy.
Annotation gap: System retrieves relevant chunks that aren't labeled. Fix with more curation rounds.
Evidence insufficiency: Corpus doesn't contain the answer. Fix with corpus expansion or query refinement.
Standard RAG evals conflate these. A low recall score could mean any of the three. Evidence-aware evaluation separates them.
For production systems:
Clinical RAG systems operate under HIPAA constraints. The evaluation protocol doesn't change compliance requirements, but it does affect audit trails:
If you're using an external LLM for evaluation, you're sending query-chunk pairs outside your perimeter. Consider local models or anonymization pipelines.
Use evidence-aware evaluation when:
Avoid it when:
The protocol adds overhead. You're tracking annotation rounds, coverage estimates, and evidence provenance. But for clinical agents and other high-stakes systems, the alternative is shipping evals that lie about system quality because they conflate retrieval performance with annotation gaps.