I wasted a week “improving” a RAG stack by reading answers and liking them. The answers got longer and more confident. Retrieval got worse.
What I measure now, in order:
Retrieval hit-rate @k
For each question I have a gold set of doc IDs (or chunk IDs) that must appear in the top-k. Hit-rate is |gold ∩ retrieved| / |gold|. If this is flat or falling, stop tuning the generator. Context precision / recall
Precision: of the chunks you stuffed into the prompt, how many were actually needed. Recall: of the needed chunks, how many you included. Easy to game precision by retrieving fewer chunks; watch recall.
Faithfulness / groundedness
Separate judge prompt (or a smaller model) that only sees answer + retrieved context and scores unsupported claims. I ban the judge from seeing the gold answer so it doesn’t reward paraphrasing the label.
Answer correctness
Only after 1–3 look okay. Otherwise you’re scoring fluent hallucinations.
Practical setup notes:
system + retrieved spans with a hash so you can replay an eval when the corpus drifts.
Made-up example from a run of 200 questions: dense@5 hit-rate 0.61, BM25@5 0.54, RRF@5 0.72. Faithfulness jumped after I cut max context from 12 chunks to 6 and required the model to cite chunk IDs — not because the LLM got “smarter,” because it had less junk to invent from.
Anyone else separating retrieval metrics from generation metrics this hard, or do you still ship on a single LLM-as-judge score?