“Evaluating RAG architectures is, however, challenging because there are several dimensions to consider.” - Es et al., RAGAS
Retrieval-Augmented Generation (RAG) combines information retrieval with a large language model. Instead of relying only on knowledge stored in model parameters, a RAG system retrieves relevant information from an external collection and provides that information to the model as context.
This architecture is useful for enterprise search, documentation assistants, customer support, internal knowledge bases, research tools, and other applications where answers should be grounded in a controlled knowledge source.
However, building a RAG pipeline is only the beginning. A system can retrieve irrelevant chunks, miss the most useful passage, or retrieve the right information but still generate an incomplete or unsupported answer. That is why evaluation must examine the retrieval and generation stages separately as well as together.
“RAG models combine pre-trained parametric and non-parametric memory for language generation.”-Lewis et al., Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks
Traditional application testing often checks whether a function returns an expected value. RAG systems are more complex because the final answer depends on several moving parts: document parsing, chunking, embeddings, retrieval, ranking, prompt construction, and generation.
A change to any one of these components can improve one type of question while making another type worse. Evaluation gives developers a repeatable way to compare versions of a RAG pipeline rather than relying on a few manually tested examples.
A useful mental model is to evaluate RAG in two layers. The first asks whether the system found the right evidence. The second asks whether the model used that evidence correctly.
Question
↓
Retriever
↓
Top-K Context
↓
Retrieval Evaluation
↓
LLM
↓
Generated Answer
↓
Answer Evaluation
Layer 1: Retrieval Evaluation
Retrieval evaluation measures whether the documents or chunks returned for a query contain the information needed to answer it. The goal is not simply to retrieve text that sounds similar to the question; the retrieved context should be useful for answering the question.
Layer 2: Answer Evaluation
Answer evaluation measures the quality of the final response. A response may be fluent and relevant to the question while still containing unsupported claims. Therefore, answer, correctness and faithfulness should be evaluated separately.
Retrieval quality is the foundation of a RAG system. If the correct information is absent from the retrieved context, the generator has less opportunity to produce a grounded answer.
For a simple evaluation, create a set of questions and identify which document or chunk is expected to contain the answer. Then compare the expected evidence with the retrieved results.
What Makes Retrieval Good?
Hit@K
Hit@K is 1 when at least one relevant document appears in the top K retrieved results; otherwise it is 0. It is useful for checking whether the retriever can find the required evidence within a practical context window.
Recall@K
Recall@K measures how many of the relevant items were retrieved within the top K results. It is especially useful when a question may require multiple pieces of evidence.
Precision@K
Precision@K measures how many of the top K retrieved items are relevant. High precision means the context contains less irrelevant material.
Mean Reciprocal Rank (MRR)
MRR rewards systems that place the first relevant result near the top. For each query, reciprocal rank is 1 divided by the position of the first relevant result; MRR is the mean across queries.
def retrieval_metrics(retrieved_names, gold_document, k):
"""Rank-based retrieval metrics for a single question.
gold_document is a single filename (single-label ground truth), or None for a question that no document can answer — those are excluded from retrieval scoring by the caller.
"""
if gold_document is None:
return None
gold_rank = None
for rank, name in enumerate(retrieved_names, start=1):
if name == gold_document:
gold_rank = rank
break
hit_at_1 = bool(retrieved_names) and retrieved_names[0] = gold_document
hit_at_k = gold_rank is not None
return {
"gold_rank": gold_rank,
"hit@1": hit_at_1,
"hit@k": hit_at_k,
"recall@k": 1.0 if hit_at_k else 0.0,
"precision@k": (1.0 / k) if hit_at_k else 0.0,
"mrr": (1.0 / gold_rank) if gold_rank else 0.0
}
Examples:
Relevant document rank = 1
Reciprocal Rank = 1 / 1 = 1.00
Relevant document rank = 3
Reciprocal Rank = 1 / 3 = 0.33
Context Relevance
Context relevance asks whether the retrieved passages actually address the question. This can be evaluated with labels, similarity methods, or an LLM-based judge. It is particularly useful when a vector search returns semantically related but practically unhelpful text.
Retrieving the correct passage does not guarantee a good answer. The generator can misunderstand the evidence, omit important details, or add information that is not supported by the context.
Faithfulness is one of the most important RAG-specific concepts. A response can be factually plausible while still failing the RAG requirement if its claims are not supported by the retrieved context.
For example, suppose the context says that employees receive 20 annual leave days. If the model answers that employees receive 20 days, the claim is grounded. If it additionally states that unused leave can always be converted to cash when the context never mentions such a policy, that additional claim should be flagged as unsupported.
“The ability of the LLM to exploit such passages in a faithful way” - Es et al., RAGAS
A reliable evaluation dataset should represent the questions users are actually expected to ask. It should not contain only easy questions that can be answered from a single obvious chunk.
Recommended Dataset Fields
{
"question": "How many annual leave days do employees get?",
"expected_document": "leave_policy.txt",
"expected_answer": "Employees receive 20 days of annual leave per year."
}
Useful fields include:
Question Categories
Consider a company leave-policy knowledge base.
Question:
How many annual leave days do employees get?
Retrieved document:
leave_policy.txt
Retrieved context:
Employees receive 20 days of annual leave per year.
Generated answer:
Employees receive 20 annual leave days per year.
This example is intentionally simple. A real evaluation set should include enough questions to expose retrieval misses, distractors, partial answers, and unsupported generation.
A Simple Results Table
Manual evaluation provides valuable judgment but can be slow when a dataset grows. LLM-based evaluation can help score properties such as relevance, correctness, and faithfulness at scale.
RAGAS is one example of a reference-free evaluation approach designed specifically for RAG pipelines. Its paper highlights the need to evaluate retrieval quality, faithful use of retrieved passages, and generation quality rather than treating the final answer as the only signal.
“We introduce RAGAs, a framework for reference-free evaluation of Retrieval Augmented Generation pipelines.” - Es et al., RAGAS
Automated evaluation should still be calibrated. Compare automated scores against human judgments on a representative sample, especially for high-impact applications.
What is RAG evaluation?
RAG evaluation is the process of measuring how effectively a RAG system retrieves useful evidence and how accurately and faithfully the language model uses that evidence to answer a question.
What is the most important retrieval metric?
There is no single best metric. Hit@K is easy to interpret, Recall@K measures coverage, Precision@K measures retrieved relevance, and MRR rewards placing the first relevant result near the top.
What is faithfulness in RAG?
Faithfulness measures whether the claims in the generated answer are supported by the retrieved context. It is a key signal for detecting unsupported generation.
Can I evaluate RAG without expected answers?
Yes. Some evaluation frameworks support reference-free evaluation by using the question, retrieved context, and generated answer. However, reference answers or verified evidence can be valuable for correctness checks.
How many questions should an evaluation dataset contain?
There is no universal number. Start with a representative set that covers the application's major question types, then expand it as new failure modes are discovered.
Should retrieval and answer quality be combined into one score?
An overall score can be useful for tracking, but the individual metrics should remain visible. A single score can hide whether the problem comes from retrieval or generation.
Can an LLM evaluate another LLM?
Yes. LLM-as-a-judge approaches are widely used for scalable evaluation, but they should be calibrated against human judgments and monitored for bias or inconsistency.
A reliable RAG system should not be judged only by whether its final answers sound convincing. The retrieval stage determines what evidence the model receives, while the generation stage determines how that evidence is transformed into an answer.
By measuring retrieval with metrics such as Hit@K, Recall@K, Precision@K, and MRR, teams can identify whether the right information is being found. By measuring correctness, relevance, completeness, and faithfulness, they can determine whether the final response is useful and grounded.
The most effective approach is to build a representative evaluation dataset, measure retrieval and answer quality separately, inspect failed cases, and rerun the evaluation whenever the RAG pipeline changes. With this discipline, RAG evaluation becomes a repeatable engineering practice rather than a subjective demo.
“Evaluating these RAG systems, however, poses unique challenges due to their hybrid structure and reliance on dynamic knowledge sources.”
- Yu et al., Evaluation of Retrieval-Augmented Generation: A Survey
About Author: Sneha Patel is a Backend Developer at AddWeb Solution, passionate about AI, Machine Learning, and Generative AI, with a focus on building scalable applications.