# How to Evaluate RAG Retrieval in 2026: Question Sets, Metrics That Mislead, and a Practical Recipe

> Source: <https://dev.to/hussnainoffical/how-to-evaluate-rag-retrieval-in-2026-question-sets-metrics-that-mislead-and-a-practical-recipe-2nh8>
> Published: 2026-11-06 05:41:05+00:00

Many RAG failures get blamed on "the LLM" when the generator never had a chance: the passage it needed was not in the context, or it was buried under near-duplicates. Retrieval deserves its own evaluation. It is also easy to get wrong, because the score depends on what you call "relevant", how you built the questions, which k you chose and how many questions you asked.

This article covers the whole chain as it stands in late 2026: labelled question sets (including synthetic ones and their traps), the metrics and where each misleads, statistical noise, lexical versus paraphrased questions, evaluating chunking, hybrid search and rerankers, LLM-as-judge and its documented biases, end-to-end metrics, current tools and benchmarks, and a recipe you can apply this week.

Numbers of my own come from small, reproducible experiments in my public lab repo, [applied-llm-lab](https://github.com/Hussnainoffical/applied-llm-lab). They show how metrics behave; they are not a benchmark.

Retrieval quality changes answer quality in ways the generator cannot fix. Google researchers defined *sufficient context* as context that "contains all the necessary information to provide a definitive answer" and found that insufficient context can make models *less* likely to abstain: for one model the incorrect-answer rate rose from 10.2% with no context to 66.1% with insufficient context [2]. A weak retriever can actively produce confident wrong answers.

Serious RAG work measures the retriever directly. Anthropic's Contextual Retrieval write-up used "1 minus recall@20" and reported failure rates falling from 5.7% to 3.7% with contextual embeddings, to 2.9% with contextual BM25 added, and to 1.9% with reranking [1], all measured before any answer was generated.

Retrieval-only metrics are also cheaper, deterministic once labels are fixed, and they localise problems: if recall@20 is fine but answers are wrong, look at the prompt or generator; if recall@20 is poor, no prompt will save you.

Every retrieval metric compares a ranked list against a set of relevant items. Before choosing a metric, decide what the unit of relevance is:

| Unit | Example label | Good for | Weakness | 
|---|---|---|---|
| Document ID | `doc_17` | Classic IR benchmarks, small docs | Says nothing about chunking | 
| Chunk ID | `doc_17#chunk_3` | Fast regression tests on one fixed index | Labels break when you re-chunk | 
| Text span / fact | "Each chunk keeps the file name and page number as metadata" | Comparing chunkers, rerankers, hybrid setups | Needs span matching logic | 
| Claims / nuggets | List of atomic facts from a reference answer | Multi-fact questions, end-to-end analysis | Needs an LLM or human to check entailment | 
| Tokens | Exact character ranges of relevant text | Measuring chunk efficiency | More engineering; unusual in tools | 

Chroma's chunking study notes that document-level benchmarks like MTEB and BEIR "cannot take chunking into account" and measures token-wise precision, recall and intersection over union instead [18]. RAGChecker works at claim level: claim recall is the share of ground-truth claims entailed by the retrieved chunks [19].

In my lab a chunk counts as the answer chunk only if it contains the *whole* answer fact. With fixed-size chunks some facts are cut in half, so for those questions *no* chunk can be relevant and the miss is guaranteed. Relevance definition and chunker interact, and the metric reflects both.

Questions come from three sources, in decreasing order of realism: real user queries from logs or tickets, subject-matter experts, and LLM-generated (synthetic) questions.

A useful record holds the question, the relevant span(s) or claims, optional graded relevance, a short reference answer, and tags such as `lexical` / `paraphrased`, `single-hop` / `multi-hop`, `has-identifier` and `unanswerable`. Tags let you see that a change helped one slice and hurt another. Label spans or claims rather than chunk IDs if you will ever compare chunkers: chunk-ID labels silently become wrong when you re-chunk.

Generation is now a standard feature. Ragas builds a knowledge graph from your documents, then synthesizers produce single-hop or multi-hop, specific or abstract queries with configurable length and style [9]. Langfuse documents routes via Ragas, DeepEval's synthesizer, or a custom loop writing one question per chunk [17]. Chroma's chunking study shows careful filtering: queries whose evidence excerpts were not exact matches in the corpus were rejected, and near-duplicate or weakly related queries were removed with embedding thresholds [18].

The research on trusting synthetic sets is nuanced:

Two more traps follow from how per-chunk generation works. A question written while looking at one chunk tends to reuse its vocabulary, which flatters lexical retrievers (measure this on your own set; see below). And it is answerable from exactly one chunk by construction, so multi-chunk and unanswerable questions are under-represented.

**Practical stance:** use synthetic questions for coverage and for comparing retriever configurations, keep a smaller human-written or log-derived set as the arbiter, and report them separately. If they disagree about the winner, believe the real one.

Here are the definitions most tools use. Let `rank` be the 1-based position of a relevant item, `k` the cutoff, and `R` the set of relevant items for one question. Every metric is averaged over questions.

| Metric | Per-question definition | Answers the question | 
|---|---|---|
| Hit rate@k (success@k) | 1 if any relevant item is in the top k, else 0 | "Did we find anything useful?" | 
| Recall@k | relevant items in top k / \|R\| | "How much of what is needed did we find?" | 
| Precision@k | relevant items in top k / k | "How much of the context is useful?" | 
| MRR@k | 1 / rank of the first relevant item (0 if none in top k) | "How high is the first useful item?" | 
| nDCG@k | DCG@k / ideal DCG@k, with DCG = Σ gain / log2(rank + 1) | "Are the most relevant items near the top?" (supports graded relevance) | 
| Context recall (LLM) | reference-answer claims supported by the retrieved context / all claims | Recall without labelled chunks | 
| Context precision (LLM) | rank-weighted precision over chunks judged relevant | Ranking quality without labelled chunks | 

Notes on origins and variants:

`DocumentRecallEvaluator` has `single_hit` (score 1 if `multi_hit` (full score only if `expected_output`, and warns that scores from different evaluation modes are not comparable [10].
**Hit rate@k with a large k hides ranking.** Hit@20 can stay flat while the relevant chunk slides from rank 1 to rank 15. That matters if the generator sees only the top 5, or handles long contexts unevenly: "Lost in the Middle" found performance "is often highest when relevant information occurs at the beginning or end of the input context" [38]. Chroma, by contrast, argues LLMs are relatively insensitive to position and down-weights nDCG-style metrics for RAG [18]. Both can hold for different models and context lengths, so report a rank-sensitive metric next to hit rate instead of picking one.

**MRR ignores everything after the first relevant item.** For a question needing three facts from three chunks, MRR rewards one at rank 1 and is blind to the other two. Use recall@k or claim-level context recall for multi-fact questions.

**Precision@k is capped by the number of relevant items.** With one relevant chunk, the best precision@5 is 0.20. Low precision matters when the extra chunks are distractors that pull the generator off course [2].

**Incomplete labels punish novel retrievers.** Unlabelled items count as irrelevant. BEIR found many datasets "subject to a lexical bias" from pooling with lexical systems; after missing TREC-COVID judgments were annotated, a dense retriever's nDCG@10 rose from 0.654 to 0.735, moving above BM25 [20]. Labels made by looking at BM25 results make new embedding models look worse.

**Chunk-level metrics are not comparable across chunkers.** Larger chunks contain more facts, so they win hit rate almost by definition while wasting context. Chroma's token-level IoU exposes this: a recursive splitter at 800 tokens with 400 overlap (the chunking settings OpenAI Assistants used by default at the time of the study) scored an IoU of 1.5, against 6.9 for the same splitter at 200 tokens with no overlap, and reducing overlap improved IoU because overlap adds redundant tokens [18].

**The same metric name gives different numbers in different tools.** Rank-weighted versus unweighted context precision, `single_hit` versus `multi_hit` recall, LLM-judged versus ID-based: pin the tool, version and variant in your eval report.

The experiments in [applied-llm-lab](https://github.com/Hussnainoffical/applied-llm-lab) build a tiny pipeline from scratch:

Day 04 added metrics: a 6-paragraph corpus about RAG pipelines and 12 questions, each asked in the document's own words ("original") and mostly in other words ("paraphrased"). The answer chunk is the one containing the whole fact. The harness core, in the same style:

``` python
def first_relevant_rank(ranked_chunks, fact, k=5):
    """1-based rank of the first top-k chunk that holds the whole fact, or None."""
    for rank, chunk in enumerate(ranked_chunks[:k], start=1):
        if fact in " ".join(chunk.split()):  # normalise whitespace before matching
            return rank
    return None

def summarize(ranks):
    """Average hit@1, hit@3 and reciprocal rank (MRR@5) over all questions."""
    n = len(ranks)
    return {
        "hit@1": sum(r == 1 for r in ranks) / n,
        "hit@3": sum(r is not None and r <= 3 for r in ranks) / n,
        "mrr@5": sum(1 / r for r in ranks if r) / n,
    }

assert summarize([1, 2, None]) == {"hit@1": 1 / 3, "hit@3": 2 / 3, "mrr@5": 0.5}
```

BM25 and TF-IDF cosine gave identical numbers in every row, so the table shows one value per row:

| Chunking | Questions | hit@1 | hit@3 | MRR@5 | 
|---|---|---|---|---|
| Fixed size, 120 chars | original | 0.67 | 0.67 | 0.67 | 
| Fixed size, 120 chars | paraphrased | 0.50 | 0.58 | 0.53 | 
| Fixed 120 + 30 overlap | original | 0.83 | 0.83 | 0.83 | 
| Fixed 120 + 30 overlap | paraphrased | 0.42 | 0.67 | 0.54 | 
| Sentence packing, max 160 | original | 1.00 | 1.00 | 1.00 | 
| Sentence packing, max 160 | paraphrased | 0.83 | 0.83 | 0.83 | 

What the toy set shows about the metrics:

Then the most important caveat.

With 12 questions, each question moves a score by 1/12 = 0.083. The gap between 0.42 and 0.50 hit@1 above is *one question*.

Confidence intervals make this concrete. Using the Wilson score interval for a proportion:

| Hits / questions | Point estimate | 95% Wilson interval | 
|---|---|---|
| 8 / 12 | 0.67 | 0.39 to 0.86 | 
| 10 / 12 | 0.83 | 0.55 to 0.95 | 
| 12 / 12 | 1.00 | 0.76 to 1.00 | 
| 83 / 100 | 0.83 | 0.74 to 0.89 | 
| 250 / 300 | 0.83 | 0.79 to 0.87 | 

The intervals for 8/12 and 12/12 overlap. Even the most dramatic result in my table, sentence packing beating fixed-size chunks 12/12 versus 8/12 on original questions, is not statistically convincing on its own: the four questions fixed-size missed were all hit by sentence packing, and none went the other way, which gives an exact two-sided McNemar (sign) test p-value of 2 × 0.5⁴ = 0.125.

Three habits fix most of this:

A paired randomization test needs only the standard library:

``` python
import random

def paired_randomization_test(a, b, trials=10_000, seed=0):
    """Two-sided p-value for mean(a) - mean(b), flipping signs of per-question differences."""
    diffs = [x - y for x, y in zip(a, b)]
    observed = abs(sum(diffs))
    rng = random.Random(seed)
    extreme = sum(
        abs(sum(d if rng.random() < 0.5 else -d for d in diffs)) >= observed
        for _ in range(trials)
    )
    return (extreme + 1) / (trials + 1)
```

Run on the per-question hit@1 scores of the sentence-packing and fixed-size rows above, it returns about 0.12, matching the exact test. If you prefer a library, `ranx` provides paired Student's t-test, Fisher's randomization test and Tukey's HSD alongside its metrics [42].

Paraphrases test the gap between how documents are written and how users ask. Lexical retrievers win on exact identifiers; BEIR found BM25 "a robust baseline" that dense models often failed to beat zero-shot [20]. Dense retrievers handle different wording but have their own limits: Weller et al. prove the number of top-k subsets a single-vector embedding can return is bounded by its dimension, and built LIMIT, on which state-of-the-art embedders fail despite very simple queries [25]. Reasoning-heavy queries break both: on BRIGHT's 1,384 real queries, a model scoring 59.0 nDCG@10 on BEIR scored 18.3, and explicit reasoning about the query added up to 12.2 points [23].

For your own set:

**Chunking.** Use span or claim labels so one question set fits every chunker. Report hit@k with a rank-sensitive metric and a cost measure (tokens sent to the generator, or token-level IoU [18]); otherwise bigger chunks always win. If you can, fix the context budget in *tokens*, not in number of chunks.

**Hybrid search.** Reciprocal rank fusion scores each document as Σ 1 / (k + rank) over the input rankings, with k = 60 fixed in a pilot study where "the choice was not critical" [41]. Fusion helps only if the inputs make *different* mistakes; my BM25 and TF-IDF ranked identically, so fusing them would add nothing. On FreshStack, a 2025 benchmark built from Stack Overflow questions and technical documentation, fusing four retrievers beat each one on most metrics and topics [24]. Report latency next to quality: the sparse half of a hybrid can help long synthetic questions but not short real ones, at several times the latency [29].

**Rerankers.** Evaluate the stages separately: the first stage needs high recall at the candidate depth you rerank (Anthropic reranked 150 candidates down to 20 [1]), the reranker needs good nDCG@k or MRR@k at the depth the generator sees. Rerankers do not always help: on FreshStack one improved some topics and lowered scores for some dense and fusion runs on others [24].

Labelling relevance by hand does not scale, so LLM judges are now everywhere: they power LLM-based context precision and recall, and they label pools for benchmarks.

The case for them is real. Thomas et al. found LLMs as accurate as human labellers in their experiments, and LLM labelling in production at Bing proved "not just more efficient but more accurate than the status quo" [30]. UMBRELA reproduced this with GPT-4o on TREC Deep Learning 2019 to 2023 and supported assessments in the TREC 2024 RAG track [31]. The biases are real too:

| Bias | Evidence | Mitigation | 
|---|---|---|
| Leniency / overrating | Consistent overrating, often with high confidence, sensitive to passage length and lexical cues [34]; GPT-4 more lenient than humans on borderline passages [27] | Calibrate on a human-labelled subset; strict, aspect-based prompts | 
| Keyword stuffing | GPT-4 with a basic prompt labelled about 26% of random nonsense passages perfectly relevant once the query string was inserted [32] | Test with adversarial passages first | 
| Instruction injection | A planted sentence claiming the passage holds the answer fooled several models [32] | Treat retrieved text as untrusted; delimit it | 
| Self-preference, circularity | Generator's model family favoured [27]; using the judging LLM as a reranker distorts rankings [33] | Judge from a different model family | 
| Position, verbosity | General LLM-judge biases [35] | Randomise order; judge one chunk at a time | 
| Prompt sensitivity | Quality "varies unpredictably with simple paraphrases" [30] | Freeze and version prompt and model | 

Alaofi et al. also found that agreement scores such as Cohen's kappa did not reliably predict how gullible a judge was [32], so "kappa looks fine" is not enough. Clarke and Dietz argue LLM-based assessment still cannot replace human assessment, especially for benchmarks meant to guide future research [33].

A practical middle ground is what ARES formalised: an automatic judge at scale, corrected by a small human-labelled set through prediction-powered inference, using "only a few hundred human annotations" [36].

Read both kinds together:

| Retrieval | Answer | Likely cause | 
|---|---|---|
| Good | Good | Working as intended | 
| Good | Bad | Generator, prompt, context ordering or context length | 
| Bad | Good | Model answered from parametric knowledge; fragile, and not grounded | 
| Bad | Bad | Fix retrieval first | 

The "bad retrieval, good answer" cell is why end-to-end metrics alone are not enough: on public-knowledge questions the generator can mask a broken retriever, which stops working the moment questions concern your private documents.

| Tool / benchmark | What it gives you for retrieval | Watch out for | 
|---|---|---|
| Ragas | LLM-based, non-LLM and ID-based context precision and recall; faithfulness; testset generation [5][6][9] | Legacy metric API slated for deprecation in 0.4 and removal in 1.0 [5]; pin versions | 
| DeepEval | Contextual precision, recall and relevancy as test-case metrics [10][11][12] | Scores from different evaluation modes are not comparable [10] | 
| Langfuse | Datasets, experiments, scores, LLM-as-judge evaluators, annotation queues for human labels; Ragas integration [16][17] | It is the platform; metric definitions come from what you plug in | 
| Haystack | Statistical MRR, MAP, nDCG and recall evaluators plus LLM-based ones [13][14][15] | Check `single_hit` versus`multi_hit` | 
| ranx | Fast IR metrics, paired significance tests, 20+ fusion methods including RRF [42] | Needs run/qrels formatted data | 
| BEIR | 18 datasets, 9 tasks, zero-shot retrieval [20] | Lexical bias in judgments | 
| MMTEB / MTEB | 500+ tasks across 250+ languages; MTEB(eng, v2) designed as zero-shot [21] | Leaderboard rank is not your corpus | 
| RTEB | Retrieval-focused, open plus private datasets to expose overfitting; NDCG@10 default; 20 languages, enterprise domains [22] | Beta | 
| BRIGHT | Reasoning-intensive retrieval [23] | Very different from FAQ-style RAG | 
| FreshStack | Technical-docs retrieval with nugget-based α-nDCG@10, coverage@20, recall@50 [24] | Five topics | 
| LIMIT | Stress test for single-vector embedding limits [25] | Deliberately simple queries; a stress test, not a target | 

Public benchmarks are for shortlisting embedding models and rerankers. RTEB exists precisely because public scores can overstate real-world performance through train-test overlap [22]. The decision should come from your own labelled set.

Next in the lab: hybrid search with reciprocal rank fusion, measured with the same Day 04 harness, and a learned-embedding retriever on the paraphrased set. Because BM25 and TF-IDF cosine ranked identically on this corpus, fusing those two alone is unlikely to change anything; the paraphrased questions, including the two abbreviation misses, are where a learned-embedding retriever should make the difference. With only 12 questions, I will read any change there as a direction, not a result.

If you evaluate retrieval in production, I would like to know which metric panel you settled on and how many questions it took before the numbers stopped moving.