An end-to-end RAG score can show that answer quality changed. By itself, it cannot tell you where to begin investigating. The evidence returned to the model may be wrong, or the model may have mishandled good evidence. One aggregate number hides that distinction and leaves retrieval, reranking, prompting, and generation competing for attention.
I use retrieval relevance and generation faithfulness as separate diagnostic boundaries. They are not an exhaustive taxonomy of RAG failures: corpus defects, bad labels, ambiguous questions, orchestration, context construction, and post-processing can also matter, sometimes at the same time. The practical value of the split is triage. It tells me which boundary to inspect first.
That is why I build the evaluation harness before, or at least alongside, the pipeline. Once the architecture has hardened, it may expose too little to compare what retrieval returned with what generation received and produced. Candidates, ranking, supplied context, and final answers are easier to make observable early than to reconstruct later.
Retrieval needs more than one number because different metrics expose different shapes of failure. Recall@K asks whether labeled evidence appeared within the cutoff. Precision@K shows how much of that result set is relevant rather than distracting. Mean reciprocal rank (MRR) emphasizes the position of the first relevant result, while normalized discounted cumulative gain (nDCG) evaluates the ranked ordering more broadly.
Read together, those metrics distinguish absence from noise and weak placement. A relevant result can appear within the cutoff and still land too low among distracting material to be useful to generation. Recall alone records its presence; precision and rank-sensitive metrics show why the returned context may still be poor.
In Strata-RAG, I use those four retrieval metrics alongside a separate generation gate. Weak retrieval results send me first to document representation, search, reranking, or the corpus. When retrieval behaves as expected and the generation gate fails, the answer-producing stage and its evaluator become the first inspection area. This routing narrows the search; failures can still span boundaries or originate in the evaluation labels.
A faithfulness judge evaluates the relationship between an answer and the context supplied to the generator. Relevance, sufficiency, and truth require separate evidence.
Suppose retrieval returns material that does not answer the user's question and the generator summarizes it accurately. A faithfulness judge may approve the answer because its claims are grounded in the supplied text. The answer remains wrong for the question.
Such a result can be consistent with the judge's assignment. Retrieval evaluation asks whether the system supplied the expected evidence; generation evaluation asks whether the answer stayed within what it was given. Reading both turns an opaque answer-quality regression into a more focused investigation.
Every retrieval metric depends on a decision about what counts as relevant evidence. Every generation check depends on a definition of acceptable support. The scorecard inherits the quality of those decisions.
For the companion generated corpus, ground truth is written during the same process that creates the documents and golden questions. At that point, the generator can record the document and field intended to supply an answer. This gives each label provenance within the generated corpus instead of requiring that provenance to be reconstructed later. The resulting labels reflect the assumptions in the generator, and questions created beside their answers may be easier for retrieval than real-user phrasing. The defensible claim is traceability inside the synthetic setting, not neutrality or real-world truth.
Task labels can fail in subtler ways. A question marked as visual may also be answerable from accompanying text, allowing a text-only system to appear to solve a vision task. The corpus includes a leak-guard check intended to catch that problem, but its method and results are not reported here. The useful design requirement is to test whether a declared modality is actually necessary, not merely to assign a modality label.
LLM judges require calibration. I once built one whose output was mostly not useful: its severity ordering ran in the wrong direction, and its top finding was a false positive. I abandoned it after repeated calibration attempts. Since then, I have treated calibration as most of the work of building a judge.
The project's generation gate sends the answer to a separate LLM with a structured JSON rubric covering faithfulness, answer relevance, and severity. The structure constrains the stated criteria and response shape. Accuracy and stability depend on separate calibration evidence because the evaluator remains a model with its own bias and variance.
The judge produces scores; separate, unit-tested code maps them to pass or fail. Given the same parsed scores, that policy function behaves deterministically and can be reviewed like other code. Variability upstream remains visible instead of being confused with a policy change.
The grading model can differ from the model used to generate answers. This avoids literal same-model grading while transferring trust to another model. For high-stakes work, I would sample judge decisions for human review and use automation for the remainder.
An eval-first system needs to represent insufficient evidence as a possible outcome. If evaluation assumes every query must receive an answer, it has no way to reward a justified refusal.
The same implementation provides an optional relevance floor after reranking. If no result clears that floor, retrieval returns an empty set and the generator refuses to answer. The threshold is tuned for each corpus and ships disabled by default; its meaning is corpus-specific. Applying the floor after reranking uses the pipeline's final ordering. No comparison with an earlier check is available here.
The demo is configured to answer everything for ease of exploration. Production evaluation may instead distinguish answers from refusals and determine which outcomes are justified by the available evidence. That labeling policy and its refusal results are not specified here, so the mechanism is an implementation pattern rather than a validated operating point.
A small, stable, single-format corpus with a handful of known questions may need only a compact golden set and direct inspection. Both a generated corpus and a judge harness carry setup costs.
By-construction ground truth becomes worth considering when a corpus is heterogeneous or its evaluation coverage must be explained, because each question can retain explicit provenance and modality requirements can be tested. A structured judge serves a different purpose: it scales the generation check, provided it is calibrated and its scores remain separate from deterministic gate logic.
The machinery can stay light or grow with the system. The diagnostic questions remain simple: did retrieval supply the expected evidence, and did generation use the supplied evidence faithfully? The answers do not identify a unique root cause. They tell you where to look first.