Evaluating RAG — retrieval + generation metrics that matter

A developer published a hands-on guide to evaluating retrieval-augmented generation (RAG) systems, arguing that most silent production decay happens in the retrieval layer rather than the generation layer. The writeup supplies runnable Python implementations of precision@k, recall@k, mean reciprocal rank and nDCG@k, along with guidance on when each metric applies and a minimum test-set recipe for catching regressions before they ship.

Most RAG systems that silently decay in production decay in the retrieval layer. The LLM output still looks fluent, so the surface tests pass. Users just notice that the answers get subtly wrong — citing the wrong clause, mixing up two customers, giving an outdated version of the policy. By the time anyone reports it, the regression has been live for weeks. Evaluation is the layer that catches those regressions before they ship. This article covers the metrics that actually tell you something, with runnable Python, and the minimum test-set recipe to get started. Prerequisite reading: the RAG pipeline walkthrough https://aiunplugged.in/blog/build-rag-pipeline-from-scratch/ ships the four-step loop this article plugs into. A RAG system has two independent failure modes, so it needs two independent sets of metrics: Evaluating only one layer misses half the problem. If retrieval is bad but generation masks it with plausible-sounding filler, the user sees "correct-looking wrong answers" and loses trust before anyone notices the metric that would have surfaced it. Of the top-K chunks the retriever returned for a question, how many are relevant? php def precision at k retrieved ids: list str , relevant ids: set str , k: int - float: top k = retrieved ids :k if not top k: return 0.0 hits = sum 1 for doc id in top k if doc id in relevant ids return hits / len top k Example: top-5 retrieved = A, B, C, D, E , relevant = {A, C} . Precision@5 = 2/5 = 0.4. Use it when the context window is limited and the LLM gets confused by irrelevant chunks. Higher precision means less noise in the generation prompt. Of all the relevant chunks that exist for a question, how many did the retriever surface in the top-K? php def recall at k retrieved ids: list str , relevant ids: set str , k: int - float: top k = set retrieved ids :k if not relevant ids: return 1.0 nothing to find — vacuously perfect hits = len top k & relevant ids return hits / len relevant ids Example: top-5 retrieved = A, B, C, D, E , relevant = {A, C, F} . Recall@5 = 2/3 ≈ 0.67 missed F . Use it when the answer requires combining multiple sources. If recall@k is low, the LLM is working with incomplete information and may fabricate the missing piece. How high up the ranking does the first relevant chunk appear? php def reciprocal rank retrieved ids: list str , relevant ids: set str - float: for i, doc id in enumerate retrieved ids : if doc id in relevant ids: return 1.0 / i + 1 return 0.0 def mean reciprocal rank evaluations: list tuple list str , set str - float: if not evaluations: return 0.0 return sum reciprocal rank r, rel for r, rel in evaluations / len evaluations Example: first relevant chunk at rank 1 → RR = 1.0; at rank 3 → RR = 1/3 ≈ 0.33; not retrieved → RR = 0. MRR is the right metric when the application only shows the top result to the user search autocomplete, "best match" chat answers . It's less useful for multi-document synthesis where several top chunks matter. The retrieval metric most forgiving of what order relevant chunks appear in — but rewards higher positions. Useful when some relevant chunks are more relevant than others graded relevance, not binary . php import math def dcg relevances: list float - float: """Discounted Cumulative Gain. relevances i = graded relevance of result i.""" return sum rel / math.log2 i + 2 for i, rel in enumerate relevances def ndcg at k retrieved ids: list str , relevance grades: dict str, float , k: int - float: """Normalised DCG@k. relevance grades = {doc id: grade}, grade = 0.""" actual = relevance grades.get doc id, 0.0 for doc id in retrieved ids :k ideal = sorted relevance grades.values , reverse=True :k if not ideal or sum ideal == 0: return 0.0 return dcg actual / dcg ideal nDCG = 1.0 means the retrieved ranking matches the ideal ranking. nDCG = 0 means nothing relevant was retrieved. A defensible default for most production RAG: track recall@k did we find all the needed context? and precision@k how much noise in that context? . Add MRR if the UI only shows the top result. Add nDCG only if relevance is actually graded, not binary — which is rare in practice. Retrieval metrics judge the retriever. Generation metrics judge whether the LLM used the retrieved context correctly. Did the generated answer stick to the retrieved context, or did it invent claims the chunks don't support? Faithfulness is the metric that catches hallucination in a RAG system where retrieval was fine. The honest way to measure faithfulness at scale is LLM-as-judge: a second model inspects the generated answer and the retrieved context, and flags claims in the answer that aren't supported by the context. Lightweight proxy for small test sets: php def claims in answer covered by context answer: str, context: str - float: """ Trivial version — split the answer into sentences and check whether each sentence has a token-overlap with the context above a threshold. Not accurate enough for production; use an LLM judge for anything serious. """ import re sentences = re.split r" ?<= . ? \s+", answer.strip if not sentences: return 1.0 context tokens = set context.lower .split covered = 0 for s in sentences: s tokens = set s.lower .split if not s tokens: continue overlap = len s tokens & context tokens / len s tokens if overlap 0.4: covered += 1 return covered / len sentences Use this proxy to spot the worst drift. For reliable faithfulness in production, use LLM-as-judge with a prompt tuned to your domain. Did the answer address the question that was asked, or did it address an adjacent question? A hallucinated answer that happens to match some retrieved context can still be irrelevant. Measured by prompting a judge model: "Given this question and this answer, rate how directly the answer responds, 1 to 5." Store the average per test query, track across releases. The two retrieval-adjacent metrics that look at how well the generation step used the context: Both are classically measured with LLM-as-judge over a test set, and are the metrics that Ragas and similar libraries report by default. A test set is a list of questions with the "correct" answer for each, plus for retrieval metrics a set of chunk IDs that should have been retrieved. The smallest useful test set has roughly 20-50 questions covering the obvious categories of query the production system will see. Minimum shape: python from dataclasses import dataclass @dataclass class EvalQuery: question: str expected answer: str relevant chunk ids: set str Chunks that contain the answer category: str Tag for slicing results — "pricing", "troubleshooting", etc. TEST SET = EvalQuery question="What is the refund window after purchase?", expected answer="7 days from the date of purchase.", relevant chunk ids={"chunk 42"}, category="policy", , EvalQuery question="Which Python version is required?", expected answer="Python 3.10 or later.", relevant chunk ids={"chunk 17", "chunk 18"}, category="setup", , ... 18 more Where the questions come from matters more than how many. Three defensible sources: The handbook's Chapter 4 covers a five-step human review protocol for assembling the first 50 queries, and the pattern for growing the test set from production logs. For a free-article minimum, 20 hand-written queries covering 3-5 categories is enough to catch obvious regressions. Wire the metrics into pytest so the eval runs on every PR. Minimum version: python import pytest from your rag app import rag pipeline your retrieve + generate function from your eval set import TEST SET from your metrics import precision at k, recall at k @pytest.fixture scope="session" def predictions : results = for query in TEST SET: retrieved ids, answer = rag pipeline query.question results.append { "query": query, "retrieved ids": retrieved ids, "answer": answer, } return results def test retrieval recall above threshold predictions : recalls = recall at k p "retrieved ids" , p "query" .relevant chunk ids, k=5 for p in predictions avg = sum recalls / len recalls assert avg 0.75, f"Average recall@5 = {avg:.3f}, below 0.75 threshold" def test retrieval precision above threshold predictions : precisions = precision at k p "retrieved ids" , p "query" .relevant chunk ids, k=5 for p in predictions avg = sum precisions / len precisions assert avg 0.30, f"Average precision@5 = {avg:.3f}, below 0.30 threshold" Thresholds 0.75 recall, 0.30 precision are starting points — tune to the system's current baseline, then enforce no regression below it. Run on every pull request so a chunking tweak or embedding model change can't silently break retrieval quality. Three evaluation libraries dominate for RAG: pip install ragas — the de facto standard for RAG-specific metrics. Ships faithfulness, answer relevance, context recall, context precision out of the box, using an LLM judge. Lowest-friction way to add generation metrics. pip install deepeval — pytest-native. Metrics look like unit tests, which is nice when CI integration is the priority. Covers RAG and non-RAG LLM apps. npm install -g promptfoo or Python API — config-driven YAML eval sets, side-by-side model comparison, caching built in. Good when the main thing being evaluated is prompt + model choice, not retrieval. Each has trade-offs around LLM-judge cost, metric calibration, and how it integrates with pytest. The handbook's Chapter 4 includes a decision table for picking between them based on team shape and existing CI. Evaluation in production has a longer tail than the metrics above. The parts not covered here, and where to find them: Those live in Chapter 4 of The RAG Production Handbook https://aiunplugged.in/products/rag-production-handbook/ — launching November 9, 2026. First 20 buyers at ₹699 regular launch ₹999, rising to ₹1,499 on Dec 1 . Related reading: