{"slug": "evaluating-rag-retrieval-generation-metrics-that-matter", "title": "Evaluating RAG — retrieval + generation metrics that matter", "summary": "A developer published a hands-on guide to evaluating retrieval-augmented generation (RAG) systems, arguing that most silent production decay happens in the retrieval layer rather than the generation layer. The writeup supplies runnable Python implementations of precision@k, recall@k, mean reciprocal rank and nDCG@k, along with guidance on when each metric applies and a minimum test-set recipe for catching regressions before they ship.", "body_md": "Most RAG systems that silently decay in production decay in the retrieval layer. The LLM output still looks fluent, so the surface tests pass. Users just notice that the answers get subtly wrong — citing the wrong clause, mixing up two customers, giving an outdated version of the policy. By the time anyone reports it, the regression has been live for weeks.\n\nEvaluation is the layer that catches those regressions before they ship. This article covers the metrics that actually tell you something, with runnable Python, and the minimum test-set recipe to get started. Prerequisite reading: the [RAG pipeline walkthrough](https://aiunplugged.in/blog/build-rag-pipeline-from-scratch/) ships the four-step loop this article plugs into.\n\nA RAG system has two independent failure modes, so it needs two independent sets of metrics:\n\nEvaluating only one layer misses half the problem. If retrieval is bad but generation masks it with plausible-sounding filler, the user sees \"correct-looking wrong answers\" and loses trust before anyone notices the metric that would have surfaced it.\n\nOf the top-K chunks the retriever returned for a question, how many are relevant?\n\n``` php\ndef precision_at_k(retrieved_ids: list[str], relevant_ids: set[str], k: int) -> float:\n    top_k = retrieved_ids[:k]\n    if not top_k:\n        return 0.0\n    hits = sum(1 for doc_id in top_k if doc_id in relevant_ids)\n    return hits / len(top_k)\n```\n\nExample: top-5 retrieved = `[A, B, C, D, E]`, relevant = `{A, C}`. Precision@5 = 2/5 = 0.4.\n\nUse it when the context window is limited and the LLM gets confused by irrelevant chunks. Higher precision means less noise in the generation prompt.\n\nOf all the relevant chunks that exist for a question, how many did the retriever surface in the top-K?\n\n``` php\ndef recall_at_k(retrieved_ids: list[str], relevant_ids: set[str], k: int) -> float:\n    top_k = set(retrieved_ids[:k])\n    if not relevant_ids:\n        return 1.0  # nothing to find — vacuously perfect\n    hits = len(top_k & relevant_ids)\n    return hits / len(relevant_ids)\n```\n\nExample: top-5 retrieved = `[A, B, C, D, E]`, relevant = `{A, C, F}`. Recall@5 = 2/3 ≈ 0.67 (missed F).\n\nUse it when the answer requires combining multiple sources. If recall@k is low, the LLM is working with incomplete information and may fabricate the missing piece.\n\nHow high up the ranking does the first relevant chunk appear?\n\n``` php\ndef reciprocal_rank(retrieved_ids: list[str], relevant_ids: set[str]) -> float:\n    for i, doc_id in enumerate(retrieved_ids):\n        if doc_id in relevant_ids:\n            return 1.0 / (i + 1)\n    return 0.0\n\ndef mean_reciprocal_rank(evaluations: list[tuple[list[str], set[str]]]) -> float:\n    if not evaluations:\n        return 0.0\n    return sum(reciprocal_rank(r, rel) for r, rel in evaluations) / len(evaluations)\n```\n\nExample: first relevant chunk at rank 1 → RR = 1.0; at rank 3 → RR = 1/3 ≈ 0.33; not retrieved → RR = 0.\n\nMRR is the right metric when the application only shows the top result to the user (search autocomplete, \"best match\" chat answers). It's less useful for multi-document synthesis where several top chunks matter.\n\nThe retrieval metric most forgiving of what order relevant chunks appear in — but rewards higher positions. Useful when some relevant chunks are more relevant than others (graded relevance, not binary).\n\n``` php\nimport math\n\ndef dcg(relevances: list[float]) -> float:\n    \"\"\"Discounted Cumulative Gain. relevances[i] = graded relevance of result i.\"\"\"\n    return sum(rel / math.log2(i + 2) for i, rel in enumerate(relevances))\n\ndef ndcg_at_k(retrieved_ids: list[str], relevance_grades: dict[str, float], k: int) -> float:\n    \"\"\"Normalised DCG@k. relevance_grades = {doc_id: grade}, grade >= 0.\"\"\"\n    actual = [relevance_grades.get(doc_id, 0.0) for doc_id in retrieved_ids[:k]]\n    ideal = sorted(relevance_grades.values(), reverse=True)[:k]\n    if not ideal or sum(ideal) == 0:\n        return 0.0\n    return dcg(actual) / dcg(ideal)\n```\n\nnDCG = 1.0 means the retrieved ranking matches the ideal ranking. nDCG = 0 means nothing relevant was retrieved.\n\nA defensible default for most production RAG: track **recall@k** (did we find all the needed context?) and **precision@k** (how much noise in that context?). Add **MRR** if the UI only shows the top result. Add **nDCG** only if relevance is actually graded, not binary — which is rare in practice.\n\nRetrieval metrics judge the retriever. Generation metrics judge whether the LLM used the retrieved context correctly.\n\nDid the generated answer stick to the retrieved context, or did it invent claims the chunks don't support? Faithfulness is the metric that catches hallucination in a RAG system where retrieval was fine.\n\nThe honest way to measure faithfulness at scale is LLM-as-judge: a second model inspects the generated answer and the retrieved context, and flags claims in the answer that aren't supported by the context. Lightweight proxy for small test sets:\n\n``` php\ndef claims_in_answer_covered_by_context(answer: str, context: str) -> float:\n    \"\"\"\n    Trivial version — split the answer into sentences and check whether each\n    sentence has a token-overlap with the context above a threshold.\n    Not accurate enough for production; use an LLM judge for anything serious.\n    \"\"\"\n    import re\n    sentences = re.split(r\"(?<=[.!?])\\s+\", answer.strip())\n    if not sentences:\n        return 1.0\n\n    context_tokens = set(context.lower().split())\n    covered = 0\n    for s in sentences:\n        s_tokens = set(s.lower().split())\n        if not s_tokens:\n            continue\n        overlap = len(s_tokens & context_tokens) / len(s_tokens)\n        if overlap > 0.4:\n            covered += 1\n    return covered / len(sentences)\n```\n\nUse this proxy to spot the worst drift. For reliable faithfulness in production, use LLM-as-judge with a prompt tuned to your domain.\n\nDid the answer address the question that was asked, or did it address an adjacent question? A hallucinated answer that happens to match some retrieved context can still be irrelevant.\n\nMeasured by prompting a judge model: \"Given this question and this answer, rate how directly the answer responds, 1 to 5.\" Store the average per test query, track across releases.\n\nThe two retrieval-adjacent metrics that look at how well the generation step used the context:\n\nBoth are classically measured with LLM-as-judge over a test set, and are the metrics that Ragas and similar libraries report by default.\n\nA test set is a list of questions with the \"correct\" answer for each, plus (for retrieval metrics) a set of chunk IDs that should have been retrieved. The smallest useful test set has roughly 20-50 questions covering the obvious categories of query the production system will see.\n\nMinimum shape:\n\n``` python\nfrom dataclasses import dataclass\n\n@dataclass\nclass EvalQuery:\n    question: str\n    expected_answer: str\n    relevant_chunk_ids: set[str]  # Chunks that contain the answer\n    category: str  # Tag for slicing results — \"pricing\", \"troubleshooting\", etc.\n\nTEST_SET = [\n    EvalQuery(\n        question=\"What is the refund window after purchase?\",\n        expected_answer=\"7 days from the date of purchase.\",\n        relevant_chunk_ids={\"chunk_42\"},\n        category=\"policy\",\n    ),\n    EvalQuery(\n        question=\"Which Python version is required?\",\n        expected_answer=\"Python 3.10 or later.\",\n        relevant_chunk_ids={\"chunk_17\", \"chunk_18\"},\n        category=\"setup\",\n    ),\n    # ... 18 more\n]\n```\n\nWhere the questions come from matters more than how many. Three defensible sources:\n\nThe handbook's Chapter 4 covers a five-step human review protocol for assembling the first 50 queries, and the pattern for growing the test set from production logs. For a free-article minimum, 20 hand-written queries covering 3-5 categories is enough to catch obvious regressions.\n\nWire the metrics into pytest so the eval runs on every PR. Minimum version:\n\n``` python\nimport pytest\n\nfrom your_rag_app import rag_pipeline  # your retrieve + generate function\nfrom your_eval_set import TEST_SET\nfrom your_metrics import precision_at_k, recall_at_k\n\n@pytest.fixture(scope=\"session\")\ndef predictions():\n    results = []\n    for query in TEST_SET:\n        retrieved_ids, answer = rag_pipeline(query.question)\n        results.append({\n            \"query\": query,\n            \"retrieved_ids\": retrieved_ids,\n            \"answer\": answer,\n        })\n    return results\n\ndef test_retrieval_recall_above_threshold(predictions):\n    recalls = [\n        recall_at_k(p[\"retrieved_ids\"], p[\"query\"].relevant_chunk_ids, k=5)\n        for p in predictions\n    ]\n    avg = sum(recalls) / len(recalls)\n    assert avg > 0.75, f\"Average recall@5 = {avg:.3f}, below 0.75 threshold\"\n\ndef test_retrieval_precision_above_threshold(predictions):\n    precisions = [\n        precision_at_k(p[\"retrieved_ids\"], p[\"query\"].relevant_chunk_ids, k=5)\n        for p in predictions\n    ]\n    avg = sum(precisions) / len(precisions)\n    assert avg > 0.30, f\"Average precision@5 = {avg:.3f}, below 0.30 threshold\"\n```\n\nThresholds (0.75 recall, 0.30 precision) are starting points — tune to the system's current baseline, then enforce no regression below it. Run on every pull request so a chunking tweak or embedding model change can't silently break retrieval quality.\n\nThree evaluation libraries dominate for RAG:\n\n`pip install ragas`) — the de facto standard for RAG-specific metrics. Ships faithfulness, answer relevance, context recall, context precision out of the box, using an LLM judge. Lowest-friction way to add generation metrics.`pip install deepeval`) — pytest-native. Metrics look like unit tests, which is nice when CI integration is the priority. Covers RAG and non-RAG LLM apps.`npm install -g promptfoo` or Python API) — config-driven YAML eval sets, side-by-side model comparison, caching built in. Good when the main thing being evaluated is prompt + model choice, not retrieval.\nEach has trade-offs around LLM-judge cost, metric calibration, and how it integrates with pytest. The handbook's Chapter 4 includes a decision table for picking between them based on team shape and existing CI.\n\nEvaluation in production has a longer tail than the metrics above. The parts not covered here, and where to find them:\n\nThose live in Chapter 4 of [The RAG Production Handbook](https://aiunplugged.in/products/rag-production-handbook/) — launching November 9, 2026. First 20 buyers at ₹699 (regular launch ₹999, rising to ₹1,499 on Dec 1).\n\nRelated reading:", "url": "https://wpnews.pro/news/evaluating-rag-retrieval-generation-metrics-that-matter", "canonical_source": "https://dev.to/aiunplugged/evaluating-rag-retrieval-generation-metrics-that-matter-2ffp", "published_at": "2026-10-24 19:05:54+00:00", "updated_at": "2026-10-24 19:12:49.130904+00:00", "lang": "en", "topics": ["ai-tools", "large-language-models", "artificial-intelligence", "developer-tools"], "entities": [], "also_reported_by": [], "alternates": {"html": "https://wpnews.pro/news/evaluating-rag-retrieval-generation-metrics-that-matter", "markdown": "https://wpnews.pro/news/evaluating-rag-retrieval-generation-metrics-that-matter.md", "text": "https://wpnews.pro/news/evaluating-rag-retrieval-generation-metrics-that-matter.txt", "jsonld": "https://wpnews.pro/news/evaluating-rag-retrieval-generation-metrics-that-matter.jsonld"}}