{"slug": "retrieval-augmented-self-recall-part-4-benchmarking-retrieval-and-honesty", "title": "Retrieval-Augmented Self-Recall — Part 4: Benchmarking Retrieval *and* Honesty", "summary": "A developer building the RE-call system created a new benchmark harness that evaluates both retrieval quality and honesty, measuring whether a system correctly abstains when no answer exists. Testing revealed that hybrid fusion and reranking provide no benefit on easy corpora with strong embedders like bge-small, while a hard-coded similarity threshold for abstention failed completely when the embedder was changed.", "body_md": "*Part 4 of Retrieval-Augmented Self-Recall. Code: RE-call. Part 3: the honesty guards.*\n\nA standard RAG benchmark would have handed my retriever a perfect score on a day it was confidently answering questions it had no data for.\n\nThat's not a bug in my system. It's a gap in what those benchmarks ask. Every one of them asks the same thing: *when there was an answer, did you rank it first?* None of them ask the question that decides whether agent memory is safe: *when there was no answer, did you say so?*\n\nSo RE-call ships its own harness. Here's how it works, and the first finding it produced.\n\nThe evaluation runs on **14 answerable queries + 5 unanswerable queries** over a synthetic corpus.\n\nThose 5 unanswerable queries are the whole reason the harness exists. They're questions the corpus genuinely cannot answer, where the *correct* behavior is to abstain — to fire `gap_warning`\n\n, not to confidently return the nearest memo. Standard retrieval benchmarks are built entirely from answerable queries; they have no way to score \"did it correctly say nothing?\" This harness is built around that case.\n\nBecause there are two jobs — rank well when there's an answer, abstain when there isn't — there are two families of metrics:\n\n**Ranking quality** (for the answerable queries):\n\n**Guard quality** (for the unanswerable queries):\n\nWhy you need both is the crux: **a system can post excellent MRR and terrible FCR.** It ranks beautifully whenever an answer exists, and lies confidently whenever one doesn't. If you only look at ranking metrics — as most RAG evals do — that failure is completely invisible. FCR is what drags it into the light.\n\nThe harness runs the full matrix: each **embedder** (`HashingEmbedder`\n\n, `bge-small`\n\n, `voyage-3`\n\n) crossed with each **fusion configuration** (dense only → hybrid → hybrid + rerank). That's what lets you answer \"which component actually earns its cost?\" instead of cargo-culting a reranker into every pipeline.\n\nAnd it runs against the real thing: **49 integration tests on a live pgvector container** (of a 150-test suite), in CI, no mock database. The benchmark exercises the actual retrieval path, not a stand-in.\n\nHere's the ablation on the **weak** (`hashing`\n\n) embedder — quality climbs monotonically as you add stages:\n\n| Configuration | MRR | nDCG@10 |\n|---|---|---|\n| Dense only | 0.63 | 0.72 |\n| + sparse (hybrid) | 0.74 | 0.80 |\n| + cross-encoder rerank | 1.00 |\n1.00 |\n\nNow the same pipeline on the **strong** `bge-small`\n\nembedder: dense retrieval *already* achieves a perfect [nDCG@10](mailto:nDCG@10). Hybrid fusion and reranking add **nothing** — there's no headroom left to recover.\n\nThe conclusion, stated plainly in the writeup: **hybrid + rerank buys the most on weaker embedders or harder corpora; on an easy corpus with a strong embedder it's redundant.**\n\nThat is a genuinely useful engineering result, because the reflex in RAG is to stack a reranker onto everything. This says: don't pay for stages your embedder has already made unnecessary. Measure first. A cross-encoder rerank on every query is real latency and real cost — and on a strong embedder over a well-covered corpus, you're buying zero.\n\nNone of these numbers come from an in-memory toy. They come from the same harness that runs in CI against real Postgres — with a dependency audit — every commit. The reason to trust the *findings* is that the *measurement* is reproducible. That's the whole pitch of this track: measure honestly, including the parts that make your work look less impressive.\n\nWhich is a good segue, because the same ablation surfaced something that made a chunk of my `gap_warning`\n\ndesign look **worthless** on certain embedders. I did not see it coming.\n\nPart 5 is the finding I keep leading with: I shipped a sensible-looking abstention threshold, switched embedders, and watched **every query that should have been refused** sail through as a confident answer. Why a hard-coded similarity threshold is a landmine — and what to do instead.\n\n(The harness itself has kept growing since this was drafted — it now also scores declared-supersession versus timestamps, and a held-out \"near-miss\" challenge set that no threshold can catch by construction. Both came out of reader comments, and both are covered in [the follow-up post](https://dev.to/gde03/retrieval-augmented-self-recall-what-the-comments-taught-me-re-call-v03-42c1).)\n\n*Part 4 of Retrieval-Augmented Self-Recall. Code: RE-call. The eval-first discipline here is the same one behind Claude Code, Beyond the Prompt.*", "url": "https://wpnews.pro/news/retrieval-augmented-self-recall-part-4-benchmarking-retrieval-and-honesty", "canonical_source": "https://dev.to/gde03/retrieval-augmented-self-recall-part-4-benchmarking-retrieval-and-honesty-aj3", "published_at": "2026-07-18 12:06:26+00:00", "updated_at": "2026-07-18 12:28:44.821453+00:00", "lang": "en", "topics": ["machine-learning", "large-language-models", "ai-research", "ai-safety", "developer-tools"], "entities": ["RE-call", "HashingEmbedder", "bge-small", "voyage-3", "pgvector"], "alternates": {"html": "https://wpnews.pro/news/retrieval-augmented-self-recall-part-4-benchmarking-retrieval-and-honesty", "markdown": "https://wpnews.pro/news/retrieval-augmented-self-recall-part-4-benchmarking-retrieval-and-honesty.md", "text": "https://wpnews.pro/news/retrieval-augmented-self-recall-part-4-benchmarking-retrieval-and-honesty.txt", "jsonld": "https://wpnews.pro/news/retrieval-augmented-self-recall-part-4-benchmarking-retrieval-and-honesty.jsonld"}}