{"slug": "rag-eval-that-doesnt-lie-to-you-hit-rate-first-then-faithfulness", "title": "RAG eval that doesn’t lie to you: hit-rate first, then faithfulness", "summary": "A practitioner's account of a RAG evaluation workflow reports that on a 200-question run, reciprocal rank fusion at k=5 reached a hit-rate of 0.72, versus 0.61 for dense retrieval at k=5 and 0.54 for BM25 at k=5. The author measures retrieval hit-rate first, then context precision and recall, then faithfulness scored by a separate judge that never sees the gold answer, and only then answer correctness, arguing that tuning the generator while retrieval is flat or falling produces fluent hallucinations. Faithfulness improved after cutting maximum context from 12 chunks to 6 and requiring the model to cite chunk IDs, which the author attributes to less junk to invent from rather than a smarter model.", "body_md": "I wasted a week “improving” a RAG stack by reading answers and liking them. The answers got longer and more confident. Retrieval got worse.\n\nWhat I measure now, in order:\n\nRetrieval hit-rate @k\n\nFor each question I have a gold set of doc IDs (or chunk IDs) that must appear in the top-k. Hit-rate is |gold ∩ retrieved| / |gold|. If this is flat or falling, stop tuning the generator.\n\nContext precision / recall\n\nPrecision: of the chunks you stuffed into the prompt, how many were actually needed. Recall: of the needed chunks, how many you included. Easy to game precision by retrieving fewer chunks; watch recall.\n\nFaithfulness / groundedness\n\nSeparate judge prompt (or a smaller model) that only sees answer + retrieved context and scores unsupported claims. I ban the judge from seeing the gold answer so it doesn’t reward paraphrasing the label.\n\nAnswer correctness\n\nOnly after 1–3 look okay. Otherwise you’re scoring fluent hallucinations.\n\nPractical setup notes:\n\n`system` + retrieved spans with a hash so you can replay an eval when the corpus drifts.\nMade-up example from a run of 200 questions: dense@5 hit-rate 0.61, BM25@5 0.54, RRF@5 0.72. Faithfulness jumped after I cut max context from 12 chunks to 6 and required the model to cite chunk IDs — not because the LLM got “smarter,” because it had less junk to invent from.\n\nAnyone else separating retrieval metrics from generation metrics this hard, or do you still ship on a single LLM-as-judge score?", "url": "https://wpnews.pro/news/rag-eval-that-doesnt-lie-to-you-hit-rate-first-then-faithfulness", "canonical_source": "https://discuss.huggingface.co/t/rag-eval-that-doesn-t-lie-to-you-hit-rate-first-then-faithfulness/191229#post_1", "published_at": "2026-10-30 12:12:18+00:00", "updated_at": "2026-10-30 12:18:23.197334+00:00", "lang": "en", "topics": ["artificial-intelligence", "large-language-models", "ai-research", "ai-tools"], "entities": [], "also_reported_by": [], "alternates": {"html": "https://wpnews.pro/news/rag-eval-that-doesnt-lie-to-you-hit-rate-first-then-faithfulness", "markdown": "https://wpnews.pro/news/rag-eval-that-doesnt-lie-to-you-hit-rate-first-then-faithfulness.md", "text": "https://wpnews.pro/news/rag-eval-that-doesnt-lie-to-you-hit-rate-first-then-faithfulness.txt", "jsonld": "https://wpnews.pro/news/rag-eval-that-doesnt-lie-to-you-hit-rate-first-then-faithfulness.jsonld"}}