# RAG eval that doesn’t lie to you: hit-rate first, then faithfulness > Source: > Published: 2026-10-30 12:12:18+00:00 I wasted a week “improving” a RAG stack by reading answers and liking them. The answers got longer and more confident. Retrieval got worse. What I measure now, in order: Retrieval hit-rate @k For each question I have a gold set of doc IDs (or chunk IDs) that must appear in the top-k. Hit-rate is |gold ∩ retrieved| / |gold|. If this is flat or falling, stop tuning the generator. Context precision / recall Precision: of the chunks you stuffed into the prompt, how many were actually needed. Recall: of the needed chunks, how many you included. Easy to game precision by retrieving fewer chunks; watch recall. Faithfulness / groundedness Separate judge prompt (or a smaller model) that only sees answer + retrieved context and scores unsupported claims. I ban the judge from seeing the gold answer so it doesn’t reward paraphrasing the label. Answer correctness Only after 1–3 look okay. Otherwise you’re scoring fluent hallucinations. Practical setup notes: `system` + retrieved spans with a hash so you can replay an eval when the corpus drifts. Made-up example from a run of 200 questions: dense@5 hit-rate 0.61, BM25@5 0.54, RRF@5 0.72. Faithfulness jumped after I cut max context from 12 chunks to 6 and required the model to cite chunk IDs — not because the LLM got “smarter,” because it had less junk to invent from. Anyone else separating retrieval metrics from generation metrics this hard, or do you still ship on a single LLM-as-judge score?