{"slug": "evidence-aware-rag-evaluation-why-ground-truth-coverage-determines-retrieval-in", "title": "Evidence-Aware RAG Evaluation: Why Ground Truth Coverage Determines Retrieval Scores in Clinical Agent Systems", "summary": "A clinical AI research team published a paper demonstrating that RAG retrieval scores are shaped by ground-truth annotation coverage, not retriever quality alone, using a breast lesion management system as a testbed. A second curation pass on an 18-document, 18-query corpus grew evidence spans from 25 to 37 and chunk targets from 36 to 51 — a 41.7% larger denominator — while Recall@3 stayed at 0.94 and retrieval rankings were unchanged. The team proposes an evidence-aware evaluation protocol that treats annotation coverage as a first-class variable and separates retrieval failures from generation failures, with a three-stage BM25-plus-dense, cross-encoder reranked pipeline at 384-token chunks and 20% overlap retrieving accepted evidence in the top three for 17 of 18 queries.", "body_md": "Your RAG system's retrieval scores depend on how much of your corpus you've annotated, not on retriever quality alone. A new paper from a clinical AI team demonstrates this measurement artifact in a breast lesion management system and proposes an evidence-aware evaluation protocol that separates annotation coverage from system quality.\n\nThe problem surfaces when you build evals for high-stakes agent systems. You can't exhaustively label every relevant passage in a large corpus. Your ground truth becomes a sample, and your retrieval metrics reflect that sample's completeness as much as your system's performance.\n\nStandard RAG evaluation assumes your ground truth annotations are complete. You label relevant chunks for each query, run your retriever, and measure precision and recall. But in practice:\n\nThe clinical team tested this with an 18-document corpus and 18 queries about breast lesion management. Initial curation produced 25 evidence spans mapped to 36 chunk targets. After a second pass, they found 37 spans and 51 chunks without changing retrieval rankings. Recall@3 stayed at 0.94, but the denominator grew by 41.7%. The system retrieved the same chunks, but the eval dataset revealed more relevant targets.\n\nIn any RAG system over a corpus larger than a few dozen documents, incomplete annotation is the norm. The paper quantifies this: a 48% increase in evidence spans (25 to 37) and a 42% increase in chunk targets (36 to 51) occurred between curation rounds, with zero change to retrieval output.\n\nThe protocol treats annotation coverage as a first-class variable. Instead of pretending your ground truth is exhaustive, you track:\n\nFor overlapping chunks, the protocol considers all chunks with equivalent evidence as relevant. If your chunking strategy creates three 384-token chunks that each contain the same critical sentence, all three count as correct retrievals.\n\nThe system uses a three-stage retriever:\n\n```\n# Pseudocode illustrating three-stage retrieval pipeline\n# Stage 1: Parallel sparse and dense retrieval\nbm25_results = bm25_index.search(query, top_k=100)\ndense_results = embedding_model.search(query, top_k=100)\n\n# Stage 2: Merge and deduplicate\n# merge_results() is a custom function that unions result sets\n# and removes duplicate chunk IDs\ncandidate_pool = merge_results(bm25_results, dense_results)\n\n# Stage 3: Cross-encoder reranking\nreranked = cross_encoder.rerank(\n    query=query,\n    candidates=candidate_pool,\n    top_k=5\n)\n```\n\nThe cross-encoder reranker consistently outperformed single-stage retrievers. With 384-token chunks and 20% overlap, it retrieved accepted evidence in the top three results for 17 of 18 queries.\n\nThe team tested chunk sizes from 128 to 1024 tokens with overlaps from 0% to 20%. Results from the paper's evaluation:\n\n| Chunk Size | Overlap | Recall@3 | R-Precision | Notes | \n|---|---|---|---|---|\n| 128 tokens | 0% | 0.78 | 0.72 | Fragments evidence spans | \n| 384 tokens | 20% | 0.94 | 0.94 | Selected configuration | \n| 512 tokens | 0% | 0.89 | 0.83 | Misses boundary cases | \n| 1024 tokens | 20% | 0.83 | 0.78 | Dilutes relevance signal | \n\nThe 384-token chunk with 20% overlap balances two failure modes. Smaller chunks fragment evidence and hurt recall. Larger chunks dilute the relevance signal and hurt precision. Overlap catches evidence that spans chunk boundaries.\n\nRetrieval is only half the system. The generation layer was evaluated separately using:\n\nThe separation matters for observability. When your agent produces a bad answer, you need to know whether retrieval failed (wrong chunks in context) or generation failed (right chunks, wrong synthesis).\n\nThe evidence-aware protocol requires tracking annotation state across curation rounds. The team's workflow:\n\nThis creates a feedback loop between system performance and ground truth quality. Your retrieval system becomes a discovery tool for improving your eval dataset.\n\nThe protocol exposes three distinct failure modes:\n\n**Retrieval failure**: System doesn't surface relevant chunks. Fix with better embeddings, reranking, or chunking strategy.\n\n**Annotation gap**: System retrieves relevant chunks that aren't labeled. Fix with more curation rounds.\n\n**Evidence insufficiency**: Corpus doesn't contain the answer. Fix with corpus expansion or query refinement.\n\nStandard RAG evals conflate these. A low recall score could mean any of the three. Evidence-aware evaluation separates them.\n\nFor production systems:\n\nClinical RAG systems operate under HIPAA constraints. The evaluation protocol doesn't change compliance requirements, but it does affect audit trails:\n\nIf you're using an external LLM for evaluation, you're sending query-chunk pairs outside your perimeter. Consider local models or anonymization pipelines.\n\n**Use evidence-aware evaluation when:**\n\n**Avoid it when:**\n\nThe protocol adds overhead. You're tracking annotation rounds, coverage estimates, and evidence provenance. But for clinical agents and other high-stakes systems, the alternative is shipping evals that lie about system quality because they conflate retrieval performance with annotation gaps.", "url": "https://wpnews.pro/news/evidence-aware-rag-evaluation-why-ground-truth-coverage-determines-retrieval-in", "canonical_source": "https://dev.to/mech_app_ai/evidence-aware-rag-evaluation-why-ground-truth-coverage-determines-retrieval-scores-in-clinical-5c6d", "published_at": "2026-10-25 20:07:10+00:00", "updated_at": "2026-10-25 20:13:32.005894+00:00", "lang": "en", "topics": ["artificial-intelligence", "ai-research", "ai-agents", "ai-safety"], "entities": [], "also_reported_by": [], "alternates": {"html": "https://wpnews.pro/news/evidence-aware-rag-evaluation-why-ground-truth-coverage-determines-retrieval-in", "markdown": "https://wpnews.pro/news/evidence-aware-rag-evaluation-why-ground-truth-coverage-determines-retrieval-in.md", "text": "https://wpnews.pro/news/evidence-aware-rag-evaluation-why-ground-truth-coverage-determines-retrieval-in.txt", "jsonld": "https://wpnews.pro/news/evidence-aware-rag-evaluation-why-ground-truth-coverage-determines-retrieval-in.jsonld"}}