Most RAG systems that silently decay in production decay in the retrieval layer. The LLM output still looks fluent, so the surface tests pass. Users just notice that the answers get subtly wrong — citing the wrong clause, mixing up two customers, giving an outdated version of the policy. By the time anyone reports it, the regression has been live for weeks.
Evaluation is the layer that catches those regressions before they ship. This article covers the metrics that actually tell you something, with runnable Python, and the minimum test-set recipe to get started. Prerequisite reading: the RAG pipeline walkthrough ships the four-step loop this article plugs into.
A RAG system has two independent failure modes, so it needs two independent sets of metrics:
Evaluating only one layer misses half the problem. If retrieval is bad but generation masks it with plausible-sounding filler, the user sees "correct-looking wrong answers" and loses trust before anyone notices the metric that would have surfaced it.
Of the top-K chunks the retriever returned for a question, how many are relevant?
def precision_at_k(retrieved_ids: list[str], relevant_ids: set[str], k: int) -> float:
top_k = retrieved_ids[:k]
if not top_k:
return 0.0
hits = sum(1 for doc_id in top_k if doc_id in relevant_ids)
return hits / len(top_k)
Example: top-5 retrieved = [A, B, C, D, E], relevant = {A, C}. Precision@5 = 2/5 = 0.4.
Use it when the context window is limited and the LLM gets confused by irrelevant chunks. Higher precision means less noise in the generation prompt.
Of all the relevant chunks that exist for a question, how many did the retriever surface in the top-K?
def recall_at_k(retrieved_ids: list[str], relevant_ids: set[str], k: int) -> float:
top_k = set(retrieved_ids[:k])
if not relevant_ids:
return 1.0 # nothing to find — vacuously perfect
hits = len(top_k & relevant_ids)
return hits / len(relevant_ids)
Example: top-5 retrieved = [A, B, C, D, E], relevant = {A, C, F}. Recall@5 = 2/3 ≈ 0.67 (missed F).
Use it when the answer requires combining multiple sources. If recall@k is low, the LLM is working with incomplete information and may fabricate the missing piece.
How high up the ranking does the first relevant chunk appear?
def reciprocal_rank(retrieved_ids: list[str], relevant_ids: set[str]) -> float:
for i, doc_id in enumerate(retrieved_ids):
if doc_id in relevant_ids:
return 1.0 / (i + 1)
return 0.0
def mean_reciprocal_rank(evaluations: list[tuple[list[str], set[str]]]) -> float:
if not evaluations:
return 0.0
return sum(reciprocal_rank(r, rel) for r, rel in evaluations) / len(evaluations)
Example: first relevant chunk at rank 1 → RR = 1.0; at rank 3 → RR = 1/3 ≈ 0.33; not retrieved → RR = 0.
MRR is the right metric when the application only shows the top result to the user (search autocomplete, "best match" chat answers). It's less useful for multi-document synthesis where several top chunks matter.
The retrieval metric most forgiving of what order relevant chunks appear in — but rewards higher positions. Useful when some relevant chunks are more relevant than others (graded relevance, not binary).
import math
def dcg(relevances: list[float]) -> float:
"""Discounted Cumulative Gain. relevances[i] = graded relevance of result i."""
return sum(rel / math.log2(i + 2) for i, rel in enumerate(relevances))
def ndcg_at_k(retrieved_ids: list[str], relevance_grades: dict[str, float], k: int) -> float:
"""Normalised DCG@k. relevance_grades = {doc_id: grade}, grade >= 0."""
actual = [relevance_grades.get(doc_id, 0.0) for doc_id in retrieved_ids[:k]]
ideal = sorted(relevance_grades.values(), reverse=True)[:k]
if not ideal or sum(ideal) == 0:
return 0.0
return dcg(actual) / dcg(ideal)
nDCG = 1.0 means the retrieved ranking matches the ideal ranking. nDCG = 0 means nothing relevant was retrieved.
A defensible default for most production RAG: track recall@k (did we find all the needed context?) and precision@k (how much noise in that context?). Add MRR if the UI only shows the top result. Add nDCG only if relevance is actually graded, not binary — which is rare in practice.
Retrieval metrics judge the retriever. Generation metrics judge whether the LLM used the retrieved context correctly.
Did the generated answer stick to the retrieved context, or did it invent claims the chunks don't support? Faithfulness is the metric that catches hallucination in a RAG system where retrieval was fine.
The honest way to measure faithfulness at scale is LLM-as-judge: a second model inspects the generated answer and the retrieved context, and flags claims in the answer that aren't supported by the context. Lightweight proxy for small test sets:
def claims_in_answer_covered_by_context(answer: str, context: str) -> float:
"""
Trivial version — split the answer into sentences and check whether each
sentence has a token-overlap with the context above a threshold.
Not accurate enough for production; use an LLM judge for anything serious.
"""
import re
sentences = re.split(r"(?<=[.!?])\s+", answer.strip())
if not sentences:
return 1.0
context_tokens = set(context.lower().split())
covered = 0
for s in sentences:
s_tokens = set(s.lower().split())
if not s_tokens:
continue
overlap = len(s_tokens & context_tokens) / len(s_tokens)
if overlap > 0.4:
covered += 1
return covered / len(sentences)
Use this proxy to spot the worst drift. For reliable faithfulness in production, use LLM-as-judge with a prompt tuned to your domain.
Did the answer address the question that was asked, or did it address an adjacent question? A hallucinated answer that happens to match some retrieved context can still be irrelevant.
Measured by prompting a judge model: "Given this question and this answer, rate how directly the answer responds, 1 to 5." Store the average per test query, track across releases.
The two retrieval-adjacent metrics that look at how well the generation step used the context:
Both are classically measured with LLM-as-judge over a test set, and are the metrics that Ragas and similar libraries report by default.
A test set is a list of questions with the "correct" answer for each, plus (for retrieval metrics) a set of chunk IDs that should have been retrieved. The smallest useful test set has roughly 20-50 questions covering the obvious categories of query the production system will see.
Minimum shape:
from dataclasses import dataclass
@dataclass
class EvalQuery:
question: str
expected_answer: str
relevant_chunk_ids: set[str] # Chunks that contain the answer
category: str # Tag for slicing results — "pricing", "troubleshooting", etc.
TEST_SET = [
EvalQuery(
question="What is the refund window after purchase?",
expected_answer="7 days from the date of purchase.",
relevant_chunk_ids={"chunk_42"},
category="policy",
),
EvalQuery(
question="Which Python version is required?",
expected_answer="Python 3.10 or later.",
relevant_chunk_ids={"chunk_17", "chunk_18"},
category="setup",
),
]
Where the questions come from matters more than how many. Three defensible sources:
The handbook's Chapter 4 covers a five-step human review protocol for assembling the first 50 queries, and the pattern for growing the test set from production logs. For a free-article minimum, 20 hand-written queries covering 3-5 categories is enough to catch obvious regressions.
Wire the metrics into pytest so the eval runs on every PR. Minimum version:
import pytest
from your_rag_app import rag_pipeline # your retrieve + generate function
from your_eval_set import TEST_SET
from your_metrics import precision_at_k, recall_at_k
@pytest.fixture(scope="session")
def predictions():
results = []
for query in TEST_SET:
retrieved_ids, answer = rag_pipeline(query.question)
results.append({
"query": query,
"retrieved_ids": retrieved_ids,
"answer": answer,
})
return results
def test_retrieval_recall_above_threshold(predictions):
recalls = [
recall_at_k(p["retrieved_ids"], p["query"].relevant_chunk_ids, k=5)
for p in predictions
]
avg = sum(recalls) / len(recalls)
assert avg > 0.75, f"Average recall@5 = {avg:.3f}, below 0.75 threshold"
def test_retrieval_precision_above_threshold(predictions):
precisions = [
precision_at_k(p["retrieved_ids"], p["query"].relevant_chunk_ids, k=5)
for p in predictions
]
avg = sum(precisions) / len(precisions)
assert avg > 0.30, f"Average precision@5 = {avg:.3f}, below 0.30 threshold"
Thresholds (0.75 recall, 0.30 precision) are starting points — tune to the system's current baseline, then enforce no regression below it. Run on every pull request so a chunking tweak or embedding model change can't silently break retrieval quality.
Three evaluation libraries dominate for RAG:
pip install ragas) — the de facto standard for RAG-specific metrics. Ships faithfulness, answer relevance, context recall, context precision out of the box, using an LLM judge. Lowest-friction way to add generation metrics.pip install deepeval) — pytest-native. Metrics look like unit tests, which is nice when CI integration is the priority. Covers RAG and non-RAG LLM apps.npm install -g promptfoo or Python API) — config-driven YAML eval sets, side-by-side model comparison, caching built in. Good when the main thing being evaluated is prompt + model choice, not retrieval.
Each has trade-offs around LLM-judge cost, metric calibration, and how it integrates with pytest. The handbook's Chapter 4 includes a decision table for picking between them based on team shape and existing CI.
Evaluation in production has a longer tail than the metrics above. The parts not covered here, and where to find them:
Those live in Chapter 4 of The RAG Production Handbook — launching November 9, 2026. First 20 buyers at ₹699 (regular launch ₹999, rising to ₹1,499 on Dec 1).
Related reading: