# RAG Evaluation: How to Know Your Retrieval Isn't Broken

> Source: <https://dev.to/mryadavgulshan/rag-evaluation-how-to-know-your-retrieval-isnt-broken-5d81>
> Published: 2026-08-29 02:30:00+00:00

*The metric taxonomy for RAG — retrieval metrics, generation metrics, the eval sets you build, and the numbers that tell you which layer is failing.*

I was reviewing a team's RAG demo at a meetup last year. The presenter asked a question, the system retrieved a chunk, and the LLM produced a fluent, well-cited answer. It looked perfect. I asked the question nobody in the room could answer: "What is your hit rate?" The presenter blinked. "What do you mean, hit rate?" We checked their logs. The retrieval was returning the correct chunk about 30% of the time — and because the demo questions were chosen for answers the system could find, nobody had noticed.

That is the whole story of RAG evaluation in one incident: fluent output hides broken retrieval. The LLM is a confident text generator, so a wrong chunk in the context produces an answer that sounds exactly as good as a right one. The only way to know which is which is to measure the layers independently, with metrics, on data you control.

I have written at length about why retrieval quality matters more than model choice. This article is the how: the metric taxonomy, the eval sets you build, the tooling that runs the numbers, and the honest rules for when this evaluation is worth the effort.

The trap is that humans are terrible at evaluating RAG output by eyeballing ten examples. Three reasons:

**Fluency is not correctness.** The model writes with equal confidence whether the retrieved chunk is right or wrong. A human reader cannot distinguish "grounded in the right source" from "plausible-sounding" without checking the source themselves — and nobody does that for every answer.

**You cannot see the failure that did not happen.** A demo question that happens to retrieve well tells you nothing about the 90% of real questions that do not. The only way to see the failure distribution is to run a fixed, labelled set repeatedly.

**The failure lives in the retrieval layer, not the output.** If retrieval returns the wrong chunk, no amount of prompt or model tuning fixes the answer. To know where to invest, you must be able to score retrieval and generation separately. A single "did the answer look good" number tells you nothing about where to fix things.

So the entire practice of RAG evaluation reduces to one question, asked with data: which layer is losing quality, retrieval or generation?

RAG evaluation splits into two families, and the biggest mistake I see is teams measuring one family and calling it done.

These score the retriever alone, before the LLM ever sees the context:

These score the LLM's output against the retrieved context:

The diagnostic logic: **if hit rate or context recall is low, fix retrieval. If faithfulness is low, your retrieval is often still fine — the problem is prompt handling, context stuffing, or the model.** These two families tell you exactly where to spend your engineering.

Metrics are meaningless without a labelled set. This is the highest-leverage work in the entire pipeline, and it is the part most teams skip because it is not glamorous. The rule I use: start with 100–200 question–document pairs built from your real logs, not from happy-path examples a developer wrote.

How to build it:

The honest advice: label 50 examples well before you label 200 badly. A clean small set beats a noisy large one, and you can grow it every week by having the team review a handful of real queries each.

You do not need to hand-score each answer. LLM-as-a-judge has matured enough that for faithfulness-type metrics it is reliable — the judge model scores whether claims are supported by context, which is a verification task, not a reasoning task. The library I reach for is RAGAS, which implements the retrieval and generation metrics above and scores a full eval set in one pass.

Here is the setup. Install, define your eval set as question/answer/context triplets, and run:

``` python
from ragas import evaluate
from ragas.metrics import (
    faithfulness,
    answer_relevancy,
    context_precision,
    context_recall,
)
from datasets import Dataset

eval_set = Dataset.from_list([
    {
        "user_input": "What is the refund window for annual plans?",
        "response": "Annual plans can be refunded within 14 days of purchase.",
        "retrieved_contexts": [
            "Annual subscriptions are eligible for a full refund within 14 days.",
        ],
        "reference": "Refund window for annual plans is 14 days.",
    },
    # ... 100+ more rows from your real logs
])

result = evaluate(
    eval_set,
    metrics=[
        faithfulness,
        answer_relevancy,
        context_precision,
        context_recall,
    ],
    llm=judge_model,  # a strong LLM acting as the judge
    embeddings=judge_embeddings,
)

result_df = result.to_pandas()
print(result_df[["context_precision", "context_recall",
                 "faithfulness", "answer_relevancy"]].mean())
```

A few honest notes on the tooling:

`reference`

field is your gold standard.Reading the output is a diagnosis, not a score. Here is the decision table I work from:

| Finding | What it means | What to fix |
|---|---|---|
| Hit rate low (<80% at k=5) | Retriever not finding the source | Chunking, embedding model, top-k, hybrid search |
| MRR low but hit rate okay | Right doc is buried | Re-ranking, larger top-k, better index params |
| Context precision low | Retrieval returns noise around the right doc | Tighter chunking, re-ranking, hybrid + fusion |
| Faithfulness low | Answer not supported by context | Context stuffing discipline, prompt hardening, model choice |
| Answer relevancy low | Not answering the question asked | Query understanding, reformulation, better instructions |

The most common real-world pattern I see: teams with high faithfulness on the demo and low hit rate on the full set. That combination is a classic — retrieval returns irrelevant chunks for most real queries, and the LLM faithfully summarizes garbage. Fix retrieval first; the generation metrics will follow.

Offline evaluation on a labelled set catches regressions you control. But it cannot catch the two failures that only appear with real traffic:

For online observability, track two lightweight numbers in production: **retrieval coverage** — the fraction of queries that return results above your relevance floor — and **feedback signals** — thumbs up/down, copy clicks, whether the user rephrased the question. Correlated with your weekly offline run, these catch the drift the labelled set cannot see.

The honest boundary, because evaluation has a cost:

The rule: evaluation effort should track the cost of being wrong. A wrong answer in a customer-facing finance assistant is worth a full harness; a wrong answer in your personal note-summarizer is worth a glance.

That meetup demo ended with a different conversation than the presenter planned. We added hit rate to their monitoring, found the 30% number, and traced it to a chunking strategy that had never been tested against their real documents. Three weeks later retrieval was above 85%, and the fluent answers were finally grounded — not because the model changed, but because someone finally measured the layer underneath it.

When someone tells you their RAG is working, ask for their hit rate. If they cannot answer, the system is not evaluated, it is demoed. Build the set, run the numbers, and let the metrics point at the layer that is actually leaking quality. That is the entire discipline, and it is the difference between a system that sounds grounded and one that is.

*Gulshan Yad