{"slug": "rag-evaluation-metrics-frameworks-and-how-to-test-a-rag-pipeline", "title": "RAG evaluation: metrics, frameworks and how to test a RAG pipeline", "summary": "A new guide on RAG evaluation argues that retrieval-augmented generation pipelines should be scored in two halves — retrieval (recall@k, precision@k, MRR, nDCG, or LLM-judged context precision and recall) and generation (faithfulness, answer relevancy, correctness) — because an end-to-end score of 70% does not reveal whether to fix the retriever or the prompt. The post compares frameworks including Ragas, DeepEval, TruLens, MLflow, Phoenix and Promptfoo, which it says agree on metric names but disagree on definitions, and cites Liu et al. (TACL 2024) finding GPT-3.5-Turbo's multi-document QA accuracy dropped by more than 20% when the answer moved to the middle of a long context, falling below its 56.1% closed-book score with 20 or 30 documents. It also includes a script, tested on a small fixture set, that compares two retrievers query by query with a confidence interval, and discloses that the author builds EvalShift, a tool for testing model changes on agents.", "body_md": "[← all posts](https://www.evalshift.dev/blog)\n\n# RAG evaluation: metrics, frameworks and how to test a RAG pipeline\n\nRAG evaluation metrics, frameworks and benchmarks compared, how far to trust an LLM judge, and how to test a retriever or model change in a RAG pipeline.\n\nRAG evaluation measures a retrieval-augmented generation pipeline in two halves, then end to end. The retrieval half asks whether the right chunks came back: recall@k, precision@k, MRR and nDCG against labeled relevant documents, or LLM-judged context precision and context recall when you have no labels. The generation half asks whether the answer used those chunks well: faithfulness (every claim supported by the retrieved context), answer relevancy (it addresses the question) and correctness against a reference answer. Score the halves separately, because an end-to-end score of 70% does not tell you whether to fix the retriever or the prompt.\n\nThe tools that compute these metrics, Ragas, DeepEval, TruLens, MLflow, Phoenix and Promptfoo among them, agree on the names and disagree on the definitions. This post covers what each metric measures and needs, how the frameworks differ, how far to trust an LLM judge for this, how to build a test set, and how to test a change to one part of the pipeline without the rest moving. It includes a short script, tested on a small fixture set, that compares two retrievers query by query with a confidence interval, a step the top-ranking guides I read leave out.\n\nA disclosure: I build EvalShift, a tool for testing model changes on agents. It has a short section at the end, which says what it does not do for RAG. Everything before it works without it.\n\n## Evaluate retrieval and generation separately\n\nA wrong RAG answer has one of three causes, and each has a different fix:\n\n| What went wrong | What you see | Where the fix is | \n|---|---|---|\n| The relevant chunk was not retrieved | Low context recall; the answer is wrong or a refusal | Chunking, embedding model, hybrid search, query rewriting, top-k | \n| It was retrieved, and the model ignored or misread it | High context recall, low faithfulness or correctness | Prompt, context order, the generator model | \n| Nothing relevant exists in the corpus | No retriever can find it; the right answer is \"I don't know\" | The corpus, and the model's willingness to abstain | \n\nThe second row is more common than it sounds. In [Lost in the Middle ↗](https://aclanthology.org/2024.tacl-1.9.pdf)\n(Liu et al., TACL 2024), models used information at the start and end of a long context much better\nthan information in the middle: GPT-3.5-Turbo's multi-document QA accuracy dropped by more than 20%\nwhen the answer moved to the middle, and with 20 or 30 documents and the answer in the middle it\nfell below its closed-book score of 56.1%. Retrieving the right chunk is necessary and not sufficient. Raising top-k improves recall\nand can lower answer quality at the same time.\n\nThe third row is the one test sets forget. If every question in your suite has an answer in the\ncorpus, you never measure whether the system admits it does not know. RGB\n([Chen et al., AAAI 2024 ↗](https://arxiv.org/abs/2309.01431)) calls this negative rejection, and\nfound models struggled with it, as with integrating information across documents and with\nretrieved text that contradicts what they know.\n\n## RAG evaluation metrics\n\n| Metric | Measures | Needs | Model calls | \n|---|---|---|---|\n| Hit rate (success@k) | Share of queries with at least one relevant chunk in the top k | Relevant chunk IDs per query | None | \n| Recall@k | Share of the relevant chunks that are in the top k | Relevant chunk IDs | None | \n| Precision@k | Share of the top k that is relevant | Relevant chunk IDs | None | \n| MRR | 1 / rank of the first relevant chunk, averaged | Relevant chunk IDs | None | \n| nDCG@k | Ranking quality with graded relevance, discounted by position | Graded relevance labels | None | \n| Context precision (LLM) | Whether relevant chunks are ranked above irrelevant ones | A reference answer | One judgment per chunk | \n| Context recall (LLM) | Share of the reference answer's claims the retrieved context supports | A reference answer | Claim extraction and checks | \n| Faithfulness, groundedness | Share of the answer's claims the retrieved context supports | The retrieved context | Claim extraction and checks | \n| Answer relevancy | Whether the answer addresses the question | Only the question and answer | One or more calls, sometimes embeddings | \n| Answer correctness | Whether the answer matches a reference | A reference answer | Claim-level comparison | \n| Noise sensitivity | How often irrelevant retrieved text makes the answer wrong | A reference answer | Claim extraction and checks | \n\n### Retrieval metrics without a model\n\nIf you can label which chunks answer each question, the classic information-retrieval metrics are\nfree, deterministic and fast. The definitions below follow the\n[ranx documentation ↗](https://amenra.github.io/ranx/metrics/):\n\n- +**Recall@k** is the relevant chunks in the top k divided by all relevant chunks for the query.\nThis is the metric to watch first, because a chunk the retriever never returned is one the\ngenerator cannot use.\n- +**Precision@k** is the relevant chunks in the top k divided by k. It is capped by how many\nrelevant chunks exist: with one relevant chunk and k = 5, the best possible score is 0.2.\n- +**Hit rate** is the share of queries with at least one relevant chunk in the top k. It is the\ncoarsest of these metrics and often the first one reported.\n- +**MRR** averages 1 divided by the rank of the first relevant chunk, so it rewards putting one\ngood chunk at the top and ignores the rest.\n- +**nDCG@k** sums each retrieved chunk's relevance grade divided by log2 of its position plus one,\nthen divides by the best possible ordering's score. It is the only one of these that uses graded\nlabels (\"answers the question\" vs \"helps\").\n\nBEIR ([Thakur et al., NeurIPS 2021 ↗](https://arxiv.org/abs/2104.08663)) reports nDCG@10 on every\ndataset because precision and recall ignore rank and MRR cannot use graded labels, and MTEB's\nretrieval tasks use the same main metric. If you report one retrieval number, report nDCG@10, and\nkeep recall@k beside it for the question \"did the chunk come back at all\".\n\nOne caution on labels: eRAG ([Salemi and Zamani, SIGIR 2024 ↗](https://arxiv.org/abs/2404.13781))\nfound that query-document relevance labels have only a small correlation with how well the RAG\nsystem answers downstream. A chunk a person marks relevant is not always a chunk the model can use.\nIts fix is to run the generator on each retrieved document alone and use the result as the label,\nwhich costs a model call per document.\n\n### LLM-judged context metrics\n\nWithout labeled chunks, a judge model decides which retrieved chunks are relevant. Ragas\n[Context Precision ↗](https://docs.ragas.io/en/stable/concepts/metrics/available_metrics/context_precision/)\njudges each chunk against a reference answer and computes a rank-weighted precision, the mean of\nprecision@k over the positions that hold a relevant chunk. Ragas\n[Context Recall ↗](https://docs.ragas.io/en/stable/concepts/metrics/available_metrics/context_recall/)\nsplits the reference answer into claims and reports the share the retrieved context supports. Both\nneed a reference answer but not chunk labels, which makes them cheaper to set up and more\nexpensive to run.\n\nRagas also has `NonLLMContextRecall` and `NonLLMContextPrecisionWithReference`, which compare\nretrieved chunks to reference chunks by string similarity, and `IDBasedContextPrecision` and\n`IDBasedContextRecall`, which compare chunk IDs. Those are the labeled metrics above under other\nnames, and they make no model call.\n\n### Generation metrics\n\n**Faithfulness** is the generation metric that matters most: the share of the answer's claims the\nretrieved context supports. It needs no reference answer, so it also works on production traffic.\nThe [hallucination detection](https://www.evalshift.dev/blog/llm-hallucination-detection) post covers the cheaper ways to\ncompute it, including small entailment models such as HHEM and MiniCheck that do the per-claim check\nwithout a large model.\n\n**Answer relevancy** checks that the answer addresses the question, not that it is right. An\nanswer can be relevant and wrong, or correct and padded with material nobody asked for.\n\n**Answer correctness** compares the answer to a reference. Ragas `FactualCorrectness` splits both\ninto claims and reports precision, recall or F1 over them. It needs reference answers, which is\nthe expensive part of a test set.\n\n**Noise sensitivity** (Ragas) is the share of the answer's claims that are incorrect, so lower is\nbetter, and a mode restricts it to errors caused by irrelevant retrieved chunks. It measures what\nhappens when you raise top-k.\n\n## Same metric name, different metric\n\nThe libraries reuse each other's names, and the definitions behind them differ in ways that move scores. Read the definition before comparing a number across tools:\n\n| Name | Ragas | DeepEval | Promptfoo | \n|---|---|---|---|\n| Faithfulness | Claims **supported** by the context / all claims | Claims that do **not contradict** the context / all claims, so an unsupported claim the context never mentions passes | `context-faithfulness` : supported claims / all claims | \n| Answer relevancy | Generates questions from the answer (3 by default) and averages their embedding similarity to the real question; the docs say the score is not guaranteed to fall between 0 and 1 | Share of the answer's statements an LLM classifies as relevant to the input; no embeddings | `answer-relevance` : generates questions from the output, compares them by embedding | \n| Context recall | Reference claims the context supports / all reference claims | Statements in `expected_output` attributable to the context / all statements | `context-recall` : attributable statements / all statements | \n| Context relevance | Not in the RAG list (the Nvidia metric `ContextRelevance` rates 0, 1 or 2 twice and averages) | `ContextualRelevancyMetric` : relevant statements in the context / all statements | `context-relevance` : required sentences / all sentences | \n\nThe faithfulness difference is the one that bites. On a support bot that adds a plausible phone number the knowledge base never mentions, Ragas Faithfulness counts that claim against the answer and DeepEval's FaithfulnessMetric does not. Neither is wrong; they answer different questions. Decide which question you are asking before you pick the metric.\n\n## Can you trust an LLM judge for RAG metrics\n\nPartly, and it depends on the metric. The research gives these numbers:\n\n- +The [RAGAS paper ↗](https://aclanthology.org/2024.eacl-demo.16.pdf) (Es et al., EACL 2024 demos)\nmeasured agreement with human pairwise judgments on its WikiEval set: 0.95 for faithfulness,\n0.78 for answer relevance and 0.70 for context relevance, which it calls \"the hardest quality\ndimension to evaluate\". The two human annotators agreed about 95% of the time on faithfulness and\ncontext relevance and 90% on answer relevance.\n- +[ARES ↗](https://aclanthology.org/2024.naacl-long.20.pdf) (Saad-Falcon et al., NAACL 2024)\nfine-tunes small judges on synthetic data and uses prediction-powered inference with about 150 or\nmore human-labeled examples to put confidence intervals on the scores. It reports evaluation\naccuracy 59.3 points above RAGAS on context relevance and 14.4 points above on answer relevance.\n- +[RAGChecker ↗](https://arxiv.org/abs/2408.08067) (Ru et al., 2024) checks answers claim by claim\nand correlated with human ratings at 0.50 to 0.62 (Pearson) across correctness, completeness and\noverall quality. Two humans correlated at 0.64 to 0.72, so the authors note \"still a clear gap\".\n- +[RAGBench ↗](https://arxiv.org/abs/2407.11005) (Friel et al., 2024) found that LLM-based RAG\nevaluation methods struggled to compete with a fine-tuned RoBERTa model.\n\nThe pattern holds across these papers: faithfulness judges are reliable, context relevance judges\nare not, and small fine-tuned classifiers compete with large judges. Before a judged metric gates\na release, label 100 to 200 of your own cases by hand and measure the judge against them, as the\n[LLM-as-a-judge](https://www.evalshift.dev/blog/llm-as-a-judge) post describes. For context relevance, prefer labeled\nrecall@k and nDCG where you can get labels.\n\n## How to build a RAG test set\n\nA RAG test case needs a question and, depending on the metrics you want, some of: the relevant documents, a reference answer, and the context your pipeline retrieved for it. Five things make a test set useful:\n\n1. +**Start from real questions.** Pull them from logs, support tickets and search queries. They\nhave the vocabulary, typos and vagueness your users actually bring.\n2. +**Label relevance at the document level, with the answer's location.** Chunk IDs change every\ntime you re-chunk, which silently invalidates chunk-level labels. Store the source document and\nthe quoted passage that answers the question, and map them to whatever chunks exist at\nevaluation time.\n3. +**Include questions the corpus cannot answer.** A share of the suite should have \"I don't know\"\nor a clarifying question as the correct output. Without them you cannot see a model that answers\neverything.\n4. +**Include the queries retrievers find hard.** Exact identifiers (error codes, SKUs, version\nnumbers), questions that need two documents, and questions phrased nothing like the document\nthat answers them.\n5. +**Use synthetic questions to fill gaps, not to replace real ones.** Ragas[`TestsetGenerator` ↗](https://docs.ragas.io/en/stable/getstarted/rag_testset_generation/) builds a\nknowledge graph from your documents and by default generates 50% single-hop and 50% multi-hop\nquestions. DeepEval's`Synthesizer` (`generate_goldens_from_docs` ) and LlamaIndex's`generate_question_context_pairs` do similar jobs. A question written by reading a chunk tends to\nreuse that chunk's wording, which flatters retrieval, so read a sample and keep real questions\nin the mix.\n\nHow many cases you need depends on how small a difference you want to detect. The\n[eval case sizing](https://www.evalshift.dev/blog/how-many-eval-cases-do-you-need) post has the arithmetic; the example below\nshows what 10 queries can and cannot tell you.\n\n## Comparing two retrievers query by query\n\nThe most common RAG change is a retrieval change: a new embedding model, a different chunk size, a reranker. Averages hide what such a change does, because it usually helps some queries and breaks others. This script scores two retrieval runs against the same labels, prints the averages, then compares recall@k query by query with a bootstrap confidence interval and lists the queries the candidate lost. It needs only the Python standard library:\n\n```\n\"\"\"Score two retrieval runs against labeled relevant chunks, and compare them query by query.\n\nUsage: python retrieval_eval.py labels.jsonl baseline.jsonl candidate.jsonl [k]\n  labels.jsonl:  {\"id\": \"q1\", \"relevant\": {\"doc-12\": 2, \"doc-40\": 1}}   grade 2 = answers it, 1 = helps\n  run files:     {\"id\": \"q1\", \"retrieved\": [\"doc-40\", \"doc-7\", \"doc-12\", ...]}   ranked, best first\n\"\"\"\nimport json\nimport math\nimport random\nimport sys\n\ndef load(path):\n    return {row[\"id\"]: row for row in map(json.loads, open(path))}\n\ndef score(retrieved, relevant, k):\n    top = retrieved[:k]\n    hits = [doc for doc in top if doc in relevant]\n    first = next((i for i, doc in enumerate(retrieved) if doc in relevant), None)\n    dcg = sum(relevant.get(doc, 0) / math.log2(i + 2) for i, doc in enumerate(top))\n    ideal = sorted(relevant.values(), reverse=True)[:k]\n    idcg = sum(grade / math.log2(i + 2) for i, grade in enumerate(ideal))\n    return {\n        f\"hit@{k}\": 1.0 if hits else 0.0,\n        f\"recall@{k}\": len(hits) / len(relevant),\n        f\"precision@{k}\": len(hits) / k,\n        \"mrr\": 0.0 if first is None else 1 / (first + 1),\n        f\"ndcg@{k}\": dcg / idcg,\n    }\n\ndef run_scores(labels, run, k):\n    return {qid: score(run[qid][\"retrieved\"], labels[qid][\"relevant\"], k) for qid in labels}\n\ndef bootstrap_ci(diffs, rounds=10_000, seed=0):\n    rng = random.Random(seed)\n    means = sorted(\n        sum(rng.choice(diffs) for _ in diffs) / len(diffs) for _ in range(rounds)\n    )\n    return means[int(0.025 * rounds)], means[int(0.975 * rounds) - 1]\n\nif __name__ == \"__main__\":\n    labels, base, cand = (load(p) for p in sys.argv[1:4])\n    k = int(sys.argv[4]) if len(sys.argv) > 4 else 5\n    a, b = run_scores(labels, base, k), run_scores(labels, cand, k)\n    print(f\"{'metric':<14}{'baseline':>10}{'candidate':>11}\")\n    for metric in a[next(iter(a))]:\n        mean_a = sum(s[metric] for s in a.values()) / len(a)\n        mean_b = sum(s[metric] for s in b.values()) / len(b)\n        print(f\"{metric:<14}{mean_a:>10.3f}{mean_b:>11.3f}\")\n    key = f\"recall@{k}\"\n    diffs = [b[q][key] - a[q][key] for q in labels]\n    low, high = bootstrap_ci(diffs)\n    better = sum(d > 0 for d in diffs)\n    worse = [q for q in labels if b[q][key] < a[q][key]]\n    print(f\"\\n{key}: candidate better on {better}, worse on {len(worse)} of {len(diffs)} queries\")\n    print(f\"mean change {sum(diffs) / len(diffs):+.3f}, 95% bootstrap CI [{low:+.3f}, {high:+.3f}]\")\n    for q in worse:\n        print(f\"  lost: {q}\")\n    sys.exit(1 if high < 0 else 0)\n```\n\nI ran it on a small fixture set: 10 support questions with graded labels, and two made-up runs standing in for the current retriever and a candidate embedding model. These are fixtures written to show the output, not measurements of real models:\n\n```\nmetric          baseline  candidate\nhit@5              0.900      0.800\nrecall@5           0.767      0.800\nprecision@5        0.260      0.280\nmrr                0.783      0.800\nndcg@5             0.672      0.800\n\nrecall@5: candidate better on 4, worse on 2 of 10 queries\nmean change +0.033, 95% bootstrap CI [-0.350, +0.383]\n  lost: err-E1042\n  lost: sku-TX-900\n```\n\nRead it from the bottom. The candidate improves nDCG@5 from 0.672 to 0.800 and recall@5 on four\nqueries, and it loses both questions that hinge on an exact identifier, an error code and a SKU.\nFour of the five averages say ship it. Only hit rate drops, from 0.900 to 0.800, because those two\nqueries now retrieve nothing relevant while one that retrieved nothing before now does. Check\nexact-identifier queries separately whenever a change reduces the weight of keyword matching,\nbecause an average lets a few wins hide them. The interval runs from\n-0.35 to +0.38, so 10 queries cannot tell you whether recall improved. The script exits 1 only\nwhen the whole interval is below zero, which makes it safe to run in CI on a suite of a few\nhundred queries, and the `lost:` lines are the cases to read either way.\n\n## RAG evaluation frameworks compared\n\nVersions and dates as of November 17, 2026:\n\n| Framework | Latest release | RAG metrics | Notes | \n|---|---|---|---|\n| [Ragas ↗](https://docs.ragas.io/en/stable/concepts/metrics/available_metrics/) | 0.4.4, Nov 14, 2026 | Context Precision, Context Recall, Context Entities Recall, Noise Sensitivity, Answer Relevancy, Faithfulness, Factual Correctness; ID-based and non-LLM variants | Test set generator; usage analytics on by default ( `RAGAS_DO_NOT_TRACK=true` ) | \n| [DeepEval ↗](https://deepeval.com/docs/metrics-faithfulness) | 4.3.4, Nov 11, 2026 | `AnswerRelevancyMetric` ,`FaithfulnessMetric` ,`ContextualPrecisionMetric` ,`ContextualRecallMetric` ,`ContextualRelevancyMetric` | Runs in pytest; every metric returns a reason; `Synthesizer` for test data | \n| [TruLens ↗](https://www.trulens.org/getting_started/core_concepts/rag_triad/) | 2.16.0, Oct 19, 2026 | The RAG triad: context relevance, groundedness, answer relevance | Published by Snowflake; OpenTelemetry tracing | \n| [MLflow ↗](https://mlflow.org/docs/latest/genai/eval-monitor/scorers/llm-judge/rag/groundedness) | 3.18.0, Nov 3, 2026 | `RetrievalRelevance` ,`RetrievalGroundedness` ,`RetrievalSufficiency` , plus`Correctness` and`RelevanceToQuery` | The retrieval judges need a traced span of type `RETRIEVER` ; Databricks uses the same judges | \n| [Arize Phoenix ↗](https://arize.com/docs/phoenix/cookbook/evaluation/evaluate-rag) | phoenix-evals 3.10.0, Nov 10, 2026 | `RetrievalRelevanceEvaluator` ,`FaithfulnessEvaluator` ,`CorrectnessEvaluator` | Its RAG cookbook computes nDCG, precision and hit rate from the relevance labels with pandas | \n| [Promptfoo ↗](https://www.promptfoo.dev/docs/configuration/expected-outputs/model-graded/) | 0.125.1, Oct 25, 2026 | `context-faithfulness` ,`context-recall` ,`context-relevance` ,`answer-relevance` | YAML assertions, default threshold 0.5 | \n| [Braintrust autoevals ↗](https://github.com/braintrustdata/autoevals) | 0.4.1, Oct 21, 2026 | `ContextPrecision` ,`ContextRecall` ,`ContextRelevancy` ,`ContextEntityRecall` ,`Faithfulness` ,`AnswerRelevancy` ,`AnswerCorrectness` | Its source says they are \"ported from the RAGAS project with some enhancements\" | \n| [LlamaIndex ↗](https://developers.llamaindex.ai/python/framework/module_guides/evaluating/) | core 0.14.26, Nov 13, 2026 | `RetrieverEvaluator` (hit rate, MRR) against expected IDs;`FaithfulnessEvaluator` ,`RelevancyEvaluator` | `generate_question_context_pairs` builds synthetic retrieval data | \n\nA note on Ragas, since \"ragas evaluation\" is one of the most searched terms in this area: after\n0.4.3 in January 2026 the repository went quiet from late February until November 13, 2026, and\n0.4.4 followed on November 14. That release fixes `import ragas` failing against recent\nlangchain-community versions and a security issue (CVE-2026-6587) in its multimodal metrics, so\nupgrade if you pinned 0.4.3. The metric API is also moving: `ragas.metrics.collections` is the\ncurrent one, and the docs say the older `ragas.metrics` classes will be removed in 1.0.\n\nWhich to pick: Ragas or DeepEval if you want the standard metrics as a Python library, DeepEval if\nyour tests already live in pytest, MLflow or Phoenix if you already trace with them, Promptfoo if\nyour evals are YAML, and plain code like the script above for labeled retrieval metrics, which need\nno framework. The [open-source evaluation frameworks](https://www.evalshift.dev/blog/open-source-llm-evaluation-frameworks)\npost compares these tools beyond RAG.\n\n## RAG benchmarks, and why they do not replace your test set\n\nPublic benchmarks are useful for choosing a component, and weak at predicting how it will do on your corpus:\n\n| Benchmark | What it contains | Use it for | \n|---|---|---|\n| [BEIR ↗](https://arxiv.org/abs/2104.08663) (NeurIPS 2021) | 18 retrieval datasets, zero-shot | Comparing retrievers out of domain; BM25 is a strong baseline | \n| [MTEB ↗](https://huggingface.co/spaces/mteb/leaderboard) ,[MMTEB ↗](https://arxiv.org/abs/2502.13595) | Embedding tasks including retrieval; MMTEB has 500+ tasks in 250+ languages | Shortlisting embedding models | \n| [BRIGHT ↗](https://arxiv.org/abs/2407.12883) (ICLR 2025) | 1,384 real queries whose relevant documents take reasoning to find | Retrieval beyond keyword and topic match | \n| [CRAG ↗](https://arxiv.org/abs/2406.04744) (NeurIPS 2024) | 4,409 questions across five domains with mock web and knowledge-graph search | End-to-end RAG with a penalty for wrong answers | \n| [RGB ↗](https://arxiv.org/abs/2309.01431) (AAAI 2024) | English and Chinese questions testing noise robustness, negative rejection, integration and counterfactuals | How the generator handles bad context | \n| [MultiHop-RAG ↗](https://arxiv.org/abs/2401.15391) (COLM 2024) | 2,556 queries over news articles, some with no answer | Questions that need several documents | \n| [FRAMES ↗](https://arxiv.org/abs/2409.12941) (NAACL 2025) | 824 questions, each needing 2 to 15 Wikipedia articles | Multi-step retrieval | \n| [RAGTruth ↗](https://aclanthology.org/2024.acl-long.585.pdf) (ACL 2024) | Nearly 18,000 RAG responses labeled for hallucination at word level | Evaluating faithfulness detectors | \n\nThree numbers from these papers make the case for testing on your own data:\n\n- +On BRIGHT, SFR-Embedding-Mistral scores nDCG@10 of 59.0 on MTEB and 18.3 on BRIGHT. A leaderboard position measures the leaderboard's queries.\n- +In CRAG, the most advanced LLMs reached at most 34% accuracy alone and 44% with straightforward RAG, and state-of-the-art industry RAG systems answered only 63% of questions without hallucinating.\n- +In MultiHop-RAG, GPT-4 scored 0.56 with the chunks its retriever found and 0.89 with the ground-truth evidence. The gap between those two numbers is the retriever's share of the error. FRAMES shows the same from the other side: accuracy went from 0.408 with no retrieval to 0.66 with multi-step retrieval.\n\n## How to test a change to a RAG pipeline\n\nMost RAG regressions arrive with a change to one component. Hold the others still and measure the one that moved:\n\n| What changed | Hold constant | Measure | \n|---|---|---|\n| Chunking, index rebuild | The labels (document plus quoted passage) | Recall@k, nDCG@10, query by query | \n| Embedding model, hybrid weights | The chunks | Recall@k, nDCG@10, and the exact-identifier queries separately | \n| Top-k, reranker | The retriever's candidates | Recall@k and faithfulness together, since more context can lower answer quality | \n| Prompt | The retrieved context, recorded per case | Faithfulness, correctness, and abstention on the unanswerable cases | \n| Generator model | The retrieved context, recorded per case | The same, plus cost and latency | \n\nHolding the context constant is the step teams skip. If you rerun retrieval while testing a new generator model, a ranking change in the vector store shows up as a model difference. Record the retrieved chunks with each test case, feed the same chunks to both models, and score retrieval in a separate run.\n\nThen compare against the current pipeline, not against an absolute threshold. A faithfulness score of 0.91 means little alone; 0.91 against 0.94 on the same cases, with an interval that excludes zero, is a regression. Gate on the metrics with explainable failures (labeled recall, correctness on cases with references, abstention on the unanswerable cases) and keep the judged metrics advisory until you have measured the judge.\n\n### When retrieval is a tool call\n\nIn an agent, retrieval is usually a tool such as `search_kb`, and the model decides whether to\nsearch, what to search for, and how many times. That adds three things to evaluate: whether it\nsearched when the answer needed it, the query it wrote, and whether the answer used what came back.\nBrowseComp-Plus ([Chen et al., 2025 ↗](https://arxiv.org/abs/2508.06600)) fixed the corpus for\ndeep-research agents to separate the two: its abstract reports GPT-5 at 55.9% accuracy, and 70.1%,\nwith fewer search calls, when paired with the Qwen3-Embedding-8B retriever. The\n[tool-call evaluation](https://www.evalshift.dev/blog/evaluating-agent-tool-calls) post covers grading the search calls\nthemselves.\n\n## Doing this with EvalShift\n\nDisclosure again: I'm the founder. [EvalShift](https://www.evalshift.dev/docs) does not compute any of the RAG metrics in\nthis post. It has no recall@k, nDCG, context precision or faithfulness score, and it does not\ncompare embedding models or rerun your retriever. For those, use the script above or one of the\nframeworks.\n\nWhat it does is the generator row of the table above: test whether a different model or a changed prompt behaves differently on recorded cases, with retrieval held constant.\n\n- +**Recorded context.** A case promoted from a[captured](https://www.evalshift.dev/docs/sdk-capture) run replays the\nrecorded model input, so if your pipeline put the retrieved chunks in the prompt, both models get\nthe same text. When retrieval is a tool,`--rounds all` feeds the recorded tool results back to\nthe candidate, so it answers from the same documents the current model saw, whatever query it\nwould have written.\n- +**Search behavior.** With retrieval as a tool,`tool_selection` checks whether the candidate still\nsearches where the current model did,`tool_arguments` compares the query it writes, and`trace_invariants` can require a search before an answer or cap the number of calls. The[evaluators docs](https://www.evalshift.dev/docs/evaluators) cover each.\n- +**The verdict.** A[`migration_policy`](https://www.evalshift.dev/docs/migration-policy) turns the differences into pass,\nconditional pass, fail or inconclusive, and inconclusive means the suite was too small to decide.\n\nThe pairwise LLM judge does not see the retrieved documents, so it cannot score groundedness. Run a faithfulness metric from one of the frameworks above on both models' answers for that.\n\n## FAQ\n\n**What is RAG evaluation?** It is measuring a retrieval-augmented generation pipeline in two\nhalves and end to end: whether the retriever returns the chunks that answer the question, measured\nwith recall@k, nDCG or LLM-judged context precision and recall, and whether the generator's answer\nis faithful to those chunks, relevant to the question and correct.\n\n**What are the main RAG evaluation metrics?** For retrieval: hit rate, recall@k, precision@k, MRR\nand nDCG@k against labeled relevant chunks, or context precision and context recall judged by an\nLLM against a reference answer. For generation: faithfulness or groundedness, answer relevancy and\nanswer correctness. Noise sensitivity measures how irrelevant retrieved text affects the answer.\n\n**What is the RAG triad?** It is TruLens's name for three reference-free RAG checks: context\nrelevance (are the retrieved chunks relevant to the query), groundedness (is each claim in the\nanswer supported by the chunks) and answer relevance (does the answer address the query).\n\n**How do you evaluate RAG without ground truth?** Use reference-free metrics: faithfulness or\ngroundedness, which checks the answer against the retrieved context, LLM-judged context relevance\nand answer relevancy. They need no reference answers, so they also run on production traffic. They\ncannot tell you whether the answer is correct, or whether a better chunk existed that was not\nretrieved.\n\n**What is the difference between Ragas and DeepEval?** Both are open-source Python libraries with\nLLM-judged RAG metrics. Ragas has more retrieval variants, including ID-based and non-LLM ones, and\na knowledge-graph test set generator. DeepEval runs in pytest and returns a reason with every\nscore. Their metrics with the same name differ: DeepEval's faithfulness counts an unsupported claim\nas faithful unless the context contradicts it, while Ragas counts it against the answer.\n\n**Is Ragas still maintained?** Yes. After a gap from late February 2026, its repository became\nactive again on November 13, 2026, and version 0.4.4 was released on November 14, 2026, with fixes\nfor import errors and a security issue in its multimodal metrics.\n\n## Keep reading\n\n- +[LLM hallucination detection](https://www.evalshift.dev/blog/llm-hallucination-detection) : faithfulness checks in depth,\nincluding small grounding models.\n- +[LLM-as-a-judge](https://www.evalshift.dev/blog/llm-as-a-judge) : how to write and validate the judge behind a RAG metric.\n- +[How many eval cases do you need](https://www.evalshift.dev/blog/how-many-eval-cases-do-you-need) : sizing a test set to\ndetect the difference you care about.\n- +[Open-source LLM evaluation frameworks](https://www.evalshift.dev/blog/open-source-llm-evaluation-frameworks) : Ragas,\nDeepEval and the rest compared beyond RAG.", "url": "https://wpnews.pro/news/rag-evaluation-metrics-frameworks-and-how-to-test-a-rag-pipeline", "canonical_source": "https://www.evalshift.dev/blog/rag-evaluation", "published_at": "2026-11-17 22:11:27.967684+00:00", "updated_at": "2026-11-17 22:11:31.050770+00:00", "lang": "en", "topics": ["ai-research", "large-language-models", "ai-tools", "mlops"], "entities": ["Ragas", "DeepEval", "TruLens", "MLflow", "Phoenix", "Promptfoo", "EvalShift", "GPT-3.5-Turbo"], "also_reported_by": [], "takeaway": "Score retrieval and generation separately — an end-to-end score of 70% cannot tell you whether to fix the retriever or the prompt — and include unanswerable questions, since GPT-3.5-Turbo's multi-document QA accuracy fell more than 20% when the answer sat mid-context, below its 56.1% closed-book sco", "recommendation": {"advice": "Split RAG evals into retrieval metrics (recall@k, MRR, nDCG) and generation metrics (faithfulness, correctness), and add unanswerable questions to test negative rejection.", "playbook": "evals-testing", "playbook_url": "https://wpnews.pro/playbooks/evals-testing"}, "alternates": {"html": "https://wpnews.pro/news/rag-evaluation-metrics-frameworks-and-how-to-test-a-rag-pipeline", "markdown": "https://wpnews.pro/news/rag-evaluation-metrics-frameworks-and-how-to-test-a-rag-pipeline.md", "text": "https://wpnews.pro/news/rag-evaluation-metrics-frameworks-and-how-to-test-a-rag-pipeline.txt", "jsonld": "https://wpnews.pro/news/rag-evaluation-metrics-frameworks-and-how-to-test-a-rag-pipeline.jsonld"}}