How to Evaluate RAG Retrieval in 2026: Question Sets, Metrics That Mislead, and a Practical Recipe

A developer published a practical guide to evaluating RAG retrieval, drawing on small reproducible experiments in a public lab repo to show how retrieval metrics behave. The writeup argues retrieval deserves its own evaluation because weak retrieval can produce confident wrong answers, citing Google research in which one model's incorrect-answer rate rose from 10.2% with no context to 66.1% with insufficient context, and Anthropic's Contextual Retrieval results, where 1-minus-recall@20 failure rates fell from 5.7% to 1.9% after contextual embeddings, BM25 and reranking. It recommends labeling text spans or claims rather than chunk IDs and tagging questions by type so changes can be assessed per slice.

Many RAG failures get blamed on "the LLM" when the generator never had a chance: the passage it needed was not in the context, or it was buried under near-duplicates. Retrieval deserves its own evaluation. It is also easy to get wrong, because the score depends on what you call "relevant", how you built the questions, which k you chose and how many questions you asked. This article covers the whole chain as it stands in late 2026: labelled question sets including synthetic ones and their traps , the metrics and where each misleads, statistical noise, lexical versus paraphrased questions, evaluating chunking, hybrid search and rerankers, LLM-as-judge and its documented biases, end-to-end metrics, current tools and benchmarks, and a recipe you can apply this week. Numbers of my own come from small, reproducible experiments in my public lab repo, applied-llm-lab https://github.com/Hussnainoffical/applied-llm-lab . They show how metrics behave; they are not a benchmark. Retrieval quality changes answer quality in ways the generator cannot fix. Google researchers defined sufficient context as context that "contains all the necessary information to provide a definitive answer" and found that insufficient context can make models less likely to abstain: for one model the incorrect-answer rate rose from 10.2% with no context to 66.1% with insufficient context 2 . A weak retriever can actively produce confident wrong answers. Serious RAG work measures the retriever directly. Anthropic's Contextual Retrieval write-up used "1 minus recall@20" and reported failure rates falling from 5.7% to 3.7% with contextual embeddings, to 2.9% with contextual BM25 added, and to 1.9% with reranking 1 , all measured before any answer was generated. Retrieval-only metrics are also cheaper, deterministic once labels are fixed, and they localise problems: if recall@20 is fine but answers are wrong, look at the prompt or generator; if recall@20 is poor, no prompt will save you. Every retrieval metric compares a ranked list against a set of relevant items. Before choosing a metric, decide what the unit of relevance is: | Unit | Example label | Good for | Weakness | |---|---|---|---| | Document ID | doc 17 | Classic IR benchmarks, small docs | Says nothing about chunking | | Chunk ID | doc 17 chunk 3 | Fast regression tests on one fixed index | Labels break when you re-chunk | | Text span / fact | "Each chunk keeps the file name and page number as metadata" | Comparing chunkers, rerankers, hybrid setups | Needs span matching logic | | Claims / nuggets | List of atomic facts from a reference answer | Multi-fact questions, end-to-end analysis | Needs an LLM or human to check entailment | | Tokens | Exact character ranges of relevant text | Measuring chunk efficiency | More engineering; unusual in tools | Chroma's chunking study notes that document-level benchmarks like MTEB and BEIR "cannot take chunking into account" and measures token-wise precision, recall and intersection over union instead 18 . RAGChecker works at claim level: claim recall is the share of ground-truth claims entailed by the retrieved chunks 19 . In my lab a chunk counts as the answer chunk only if it contains the whole answer fact. With fixed-size chunks some facts are cut in half, so for those questions no chunk can be relevant and the miss is guaranteed. Relevance definition and chunker interact, and the metric reflects both. Questions come from three sources, in decreasing order of realism: real user queries from logs or tickets, subject-matter experts, and LLM-generated synthetic questions. A useful record holds the question, the relevant span s or claims, optional graded relevance, a short reference answer, and tags such as lexical / paraphrased , single-hop / multi-hop , has-identifier and unanswerable . Tags let you see that a change helped one slice and hurt another. Label spans or claims rather than chunk IDs if you will ever compare chunkers: chunk-ID labels silently become wrong when you re-chunk. Generation is now a standard feature. Ragas builds a knowledge graph from your documents, then synthesizers produce single-hop or multi-hop, specific or abstract queries with configurable length and style 9 . Langfuse documents routes via Ragas, DeepEval's synthesizer, or a custom loop writing one question per chunk 17 . Chroma's chunking study shows careful filtering: queries whose evidence excerpts were not exact matches in the corpus were rejected, and near-duplicate or weakly related queries were removed with embedding thresholds 18 . The research on trusting synthetic sets is nuanced: Two more traps follow from how per-chunk generation works. A question written while looking at one chunk tends to reuse its vocabulary, which flatters lexical retrievers measure this on your own set; see below . And it is answerable from exactly one chunk by construction, so multi-chunk and unanswerable questions are under-represented. Practical stance: use synthetic questions for coverage and for comparing retriever configurations, keep a smaller human-written or log-derived set as the arbiter, and report them separately. If they disagree about the winner, believe the real one. Here are the definitions most tools use. Let rank be the 1-based position of a relevant item, k the cutoff, and R the set of relevant items for one question. Every metric is averaged over questions. | Metric | Per-question definition | Answers the question | |---|---|---| | Hit rate@k success@k | 1 if any relevant item is in the top k, else 0 | "Did we find anything useful?" | | Recall@k | relevant items in top k / \|R\| | "How much of what is needed did we find?" | | Precision@k | relevant items in top k / k | "How much of the context is useful?" | | MRR@k | 1 / rank of the first relevant item 0 if none in top k | "How high is the first useful item?" | | nDCG@k | DCG@k / ideal DCG@k, with DCG = Σ gain / log2 rank + 1 | "Are the most relevant items near the top?" supports graded relevance | | Context recall LLM | reference-answer claims supported by the retrieved context / all claims | Recall without labelled chunks | | Context precision LLM | rank-weighted precision over chunks judged relevant | Ranking quality without labelled chunks | Notes on origins and variants: DocumentRecallEvaluator has single hit score 1 if multi hit full score only if expected output , and warns that scores from different evaluation modes are not comparable 10 . Hit rate@k with a large k hides ranking. Hit@20 can stay flat while the relevant chunk slides from rank 1 to rank 15. That matters if the generator sees only the top 5, or handles long contexts unevenly: "Lost in the Middle" found performance "is often highest when relevant information occurs at the beginning or end of the input context" 38 . Chroma, by contrast, argues LLMs are relatively insensitive to position and down-weights nDCG-style metrics for RAG 18 . Both can hold for different models and context lengths, so report a rank-sensitive metric next to hit rate instead of picking one. MRR ignores everything after the first relevant item. For a question needing three facts from three chunks, MRR rewards one at rank 1 and is blind to the other two. Use recall@k or claim-level context recall for multi-fact questions. Precision@k is capped by the number of relevant items. With one relevant chunk, the best precision@5 is 0.20. Low precision matters when the extra chunks are distractors that pull the generator off course 2 . Incomplete labels punish novel retrievers. Unlabelled items count as irrelevant. BEIR found many datasets "subject to a lexical bias" from pooling with lexical systems; after missing TREC-COVID judgments were annotated, a dense retriever's nDCG@10 rose from 0.654 to 0.735, moving above BM25 20 . Labels made by looking at BM25 results make new embedding models look worse. Chunk-level metrics are not comparable across chunkers. Larger chunks contain more facts, so they win hit rate almost by definition while wasting context. Chroma's token-level IoU exposes this: a recursive splitter at 800 tokens with 400 overlap the chunking settings OpenAI Assistants used by default at the time of the study scored an IoU of 1.5, against 6.9 for the same splitter at 200 tokens with no overlap, and reducing overlap improved IoU because overlap adds redundant tokens 18 . The same metric name gives different numbers in different tools. Rank-weighted versus unweighted context precision, single hit versus multi hit recall, LLM-judged versus ID-based: pin the tool, version and variant in your eval report. The experiments in applied-llm-lab https://github.com/Hussnainoffical/applied-llm-lab build a tiny pipeline from scratch: Day 04 added metrics: a 6-paragraph corpus about RAG pipelines and 12 questions, each asked in the document's own words "original" and mostly in other words "paraphrased" . The answer chunk is the one containing the whole fact. The harness core, in the same style: python def first relevant rank ranked chunks, fact, k=5 : """1-based rank of the first top-k chunk that holds the whole fact, or None.""" for rank, chunk in enumerate ranked chunks :k , start=1 : if fact in " ".join chunk.split : normalise whitespace before matching return rank return None def summarize ranks : """Average hit@1, hit@3 and reciprocal rank MRR@5 over all questions.""" n = len ranks return { "hit@1": sum r == 1 for r in ranks / n, "hit@3": sum r is not None and r <= 3 for r in ranks / n, "mrr@5": sum 1 / r for r in ranks if r / n, } assert summarize 1, 2, None == {"hit@1": 1 / 3, "hit@3": 2 / 3, "mrr@5": 0.5} BM25 and TF-IDF cosine gave identical numbers in every row, so the table shows one value per row: | Chunking | Questions | hit@1 | hit@3 | MRR@5 | |---|---|---|---|---| | Fixed size, 120 chars | original | 0.67 | 0.67 | 0.67 | | Fixed size, 120 chars | paraphrased | 0.50 | 0.58 | 0.53 | | Fixed 120 + 30 overlap | original | 0.83 | 0.83 | 0.83 | | Fixed 120 + 30 overlap | paraphrased | 0.42 | 0.67 | 0.54 | | Sentence packing, max 160 | original | 1.00 | 1.00 | 1.00 | | Sentence packing, max 160 | paraphrased | 0.83 | 0.83 | 0.83 | What the toy set shows about the metrics: Then the most important caveat. With 12 questions, each question moves a score by 1/12 = 0.083. The gap between 0.42 and 0.50 hit@1 above is one question . Confidence intervals make this concrete. Using the Wilson score interval for a proportion: | Hits / questions | Point estimate | 95% Wilson interval | |---|---|---| | 8 / 12 | 0.67 | 0.39 to 0.86 | | 10 / 12 | 0.83 | 0.55 to 0.95 | | 12 / 12 | 1.00 | 0.76 to 1.00 | | 83 / 100 | 0.83 | 0.74 to 0.89 | | 250 / 300 | 0.83 | 0.79 to 0.87 | The intervals for 8/12 and 12/12 overlap. Even the most dramatic result in my table, sentence packing beating fixed-size chunks 12/12 versus 8/12 on original questions, is not statistically convincing on its own: the four questions fixed-size missed were all hit by sentence packing, and none went the other way, which gives an exact two-sided McNemar sign test p-value of 2 × 0.5⁴ = 0.125. Three habits fix most of this: A paired randomization test needs only the standard library: python import random def paired randomization test a, b, trials=10 000, seed=0 : """Two-sided p-value for mean a - mean b , flipping signs of per-question differences.""" diffs = x - y for x, y in zip a, b observed = abs sum diffs rng = random.Random seed extreme = sum abs sum d if rng.random < 0.5 else -d for d in diffs = observed for in range trials return extreme + 1 / trials + 1 Run on the per-question hit@1 scores of the sentence-packing and fixed-size rows above, it returns about 0.12, matching the exact test. If you prefer a library, ranx provides paired Student's t-test, Fisher's randomization test and Tukey's HSD alongside its metrics 42 . Paraphrases test the gap between how documents are written and how users ask. Lexical retrievers win on exact identifiers; BEIR found BM25 "a robust baseline" that dense models often failed to beat zero-shot 20 . Dense retrievers handle different wording but have their own limits: Weller et al. prove the number of top-k subsets a single-vector embedding can return is bounded by its dimension, and built LIMIT, on which state-of-the-art embedders fail despite very simple queries 25 . Reasoning-heavy queries break both: on BRIGHT's 1,384 real queries, a model scoring 59.0 nDCG@10 on BEIR scored 18.3, and explicit reasoning about the query added up to 12.2 points 23 . For your own set: Chunking. Use span or claim labels so one question set fits every chunker. Report hit@k with a rank-sensitive metric and a cost measure tokens sent to the generator, or token-level IoU 18 ; otherwise bigger chunks always win. If you can, fix the context budget in tokens , not in number of chunks. Hybrid search. Reciprocal rank fusion scores each document as Σ 1 / k + rank over the input rankings, with k = 60 fixed in a pilot study where "the choice was not critical" 41 . Fusion helps only if the inputs make different mistakes; my BM25 and TF-IDF ranked identically, so fusing them would add nothing. On FreshStack, a 2025 benchmark built from Stack Overflow questions and technical documentation, fusing four retrievers beat each one on most metrics and topics 24 . Report latency next to quality: the sparse half of a hybrid can help long synthetic questions but not short real ones, at several times the latency 29 . Rerankers. Evaluate the stages separately: the first stage needs high recall at the candidate depth you rerank Anthropic reranked 150 candidates down to 20 1 , the reranker needs good nDCG@k or MRR@k at the depth the generator sees. Rerankers do not always help: on FreshStack one improved some topics and lowered scores for some dense and fusion runs on others 24 . Labelling relevance by hand does not scale, so LLM judges are now everywhere: they power LLM-based context precision and recall, and they label pools for benchmarks. The case for them is real. Thomas et al. found LLMs as accurate as human labellers in their experiments, and LLM labelling in production at Bing proved "not just more efficient but more accurate than the status quo" 30 . UMBRELA reproduced this with GPT-4o on TREC Deep Learning 2019 to 2023 and supported assessments in the TREC 2024 RAG track 31 . The biases are real too: | Bias | Evidence | Mitigation | |---|---|---| | Leniency / overrating | Consistent overrating, often with high confidence, sensitive to passage length and lexical cues 34 ; GPT-4 more lenient than humans on borderline passages 27 | Calibrate on a human-labelled subset; strict, aspect-based prompts | | Keyword stuffing | GPT-4 with a basic prompt labelled about 26% of random nonsense passages perfectly relevant once the query string was inserted 32 | Test with adversarial passages first | | Instruction injection | A planted sentence claiming the passage holds the answer fooled several models 32 | Treat retrieved text as untrusted; delimit it | | Self-preference, circularity | Generator's model family favoured 27 ; using the judging LLM as a reranker distorts rankings 33 | Judge from a different model family | | Position, verbosity | General LLM-judge biases 35 | Randomise order; judge one chunk at a time | | Prompt sensitivity | Quality "varies unpredictably with simple paraphrases" 30 | Freeze and version prompt and model | Alaofi et al. also found that agreement scores such as Cohen's kappa did not reliably predict how gullible a judge was 32 , so "kappa looks fine" is not enough. Clarke and Dietz argue LLM-based assessment still cannot replace human assessment, especially for benchmarks meant to guide future research 33 . A practical middle ground is what ARES formalised: an automatic judge at scale, corrected by a small human-labelled set through prediction-powered inference, using "only a few hundred human annotations" 36 . Read both kinds together: | Retrieval | Answer | Likely cause | |---|---|---| | Good | Good | Working as intended | | Good | Bad | Generator, prompt, context ordering or context length | | Bad | Good | Model answered from parametric knowledge; fragile, and not grounded | | Bad | Bad | Fix retrieval first | The "bad retrieval, good answer" cell is why end-to-end metrics alone are not enough: on public-knowledge questions the generator can mask a broken retriever, which stops working the moment questions concern your private documents. | Tool / benchmark | What it gives you for retrieval | Watch out for | |---|---|---| | Ragas | LLM-based, non-LLM and ID-based context precision and recall; faithfulness; testset generation 5 6 9 | Legacy metric API slated for deprecation in 0.4 and removal in 1.0 5 ; pin versions | | DeepEval | Contextual precision, recall and relevancy as test-case metrics 10 11 12 | Scores from different evaluation modes are not comparable 10 | | Langfuse | Datasets, experiments, scores, LLM-as-judge evaluators, annotation queues for human labels; Ragas integration 16 17 | It is the platform; metric definitions come from what you plug in | | Haystack | Statistical MRR, MAP, nDCG and recall evaluators plus LLM-based ones 13 14 15 | Check single hit versus multi hit | | ranx | Fast IR metrics, paired significance tests, 20+ fusion methods including RRF 42 | Needs run/qrels formatted data | | BEIR | 18 datasets, 9 tasks, zero-shot retrieval 20 | Lexical bias in judgments | | MMTEB / MTEB | 500+ tasks across 250+ languages; MTEB eng, v2 designed as zero-shot 21 | Leaderboard rank is not your corpus | | RTEB | Retrieval-focused, open plus private datasets to expose overfitting; NDCG@10 default; 20 languages, enterprise domains 22 | Beta | | BRIGHT | Reasoning-intensive retrieval 23 | Very different from FAQ-style RAG | | FreshStack | Technical-docs retrieval with nugget-based α-nDCG@10, coverage@20, recall@50 24 | Five topics | | LIMIT | Stress test for single-vector embedding limits 25 | Deliberately simple queries; a stress test, not a target | Public benchmarks are for shortlisting embedding models and rerankers. RTEB exists precisely because public scores can overstate real-world performance through train-test overlap 22 . The decision should come from your own labelled set. Next in the lab: hybrid search with reciprocal rank fusion, measured with the same Day 04 harness, and a learned-embedding retriever on the paraphrased set. Because BM25 and TF-IDF cosine ranked identically on this corpus, fusing those two alone is unlikely to change anything; the paraphrased questions, including the two abbreviation misses, are where a learned-embedding retriever should make the difference. With only 12 questions, I will read any change there as a direction, not a result. If you evaluate retrieval in production, I would like to know which metric panel you settled on and how many questions it took before the numbers stopped moving.