# Retrieval-Augmented Self-Recall — Part 4: Benchmarking Retrieval *and* Honesty

> Source: <https://dev.to/gde03/retrieval-augmented-self-recall-part-4-benchmarking-retrieval-and-honesty-aj3>
> Published: 2026-07-18 12:06:26+00:00

*Part 4 of Retrieval-Augmented Self-Recall. Code: RE-call. Part 3: the honesty guards.*

A standard RAG benchmark would have handed my retriever a perfect score on a day it was confidently answering questions it had no data for.

That's not a bug in my system. It's a gap in what those benchmarks ask. Every one of them asks the same thing: *when there was an answer, did you rank it first?* None of them ask the question that decides whether agent memory is safe: *when there was no answer, did you say so?*

So RE-call ships its own harness. Here's how it works, and the first finding it produced.

The evaluation runs on **14 answerable queries + 5 unanswerable queries** over a synthetic corpus.

Those 5 unanswerable queries are the whole reason the harness exists. They're questions the corpus genuinely cannot answer, where the *correct* behavior is to abstain — to fire `gap_warning`

, not to confidently return the nearest memo. Standard retrieval benchmarks are built entirely from answerable queries; they have no way to score "did it correctly say nothing?" This harness is built around that case.

Because there are two jobs — rank well when there's an answer, abstain when there isn't — there are two families of metrics:

**Ranking quality** (for the answerable queries):

**Guard quality** (for the unanswerable queries):

Why you need both is the crux: **a system can post excellent MRR and terrible FCR.** It ranks beautifully whenever an answer exists, and lies confidently whenever one doesn't. If you only look at ranking metrics — as most RAG evals do — that failure is completely invisible. FCR is what drags it into the light.

The harness runs the full matrix: each **embedder** (`HashingEmbedder`

, `bge-small`

, `voyage-3`

) crossed with each **fusion configuration** (dense only → hybrid → hybrid + rerank). That's what lets you answer "which component actually earns its cost?" instead of cargo-culting a reranker into every pipeline.

And it runs against the real thing: **49 integration tests on a live pgvector container** (of a 150-test suite), in CI, no mock database. The benchmark exercises the actual retrieval path, not a stand-in.

Here's the ablation on the **weak** (`hashing`

) embedder — quality climbs monotonically as you add stages:

| Configuration | MRR | nDCG@10 |
|---|---|---|
| Dense only | 0.63 | 0.72 |
| + sparse (hybrid) | 0.74 | 0.80 |
| + cross-encoder rerank | 1.00 |
1.00 |

Now the same pipeline on the **strong** `bge-small`

embedder: dense retrieval *already* achieves a perfect [nDCG@10](mailto:nDCG@10). Hybrid fusion and reranking add **nothing** — there's no headroom left to recover.

The conclusion, stated plainly in the writeup: **hybrid + rerank buys the most on weaker embedders or harder corpora; on an easy corpus with a strong embedder it's redundant.**

That is a genuinely useful engineering result, because the reflex in RAG is to stack a reranker onto everything. This says: don't pay for stages your embedder has already made unnecessary. Measure first. A cross-encoder rerank on every query is real latency and real cost — and on a strong embedder over a well-covered corpus, you're buying zero.

None of these numbers come from an in-memory toy. They come from the same harness that runs in CI against real Postgres — with a dependency audit — every commit. The reason to trust the *findings* is that the *measurement* is reproducible. That's the whole pitch of this track: measure honestly, including the parts that make your work look less impressive.

Which is a good segue, because the same ablation surfaced something that made a chunk of my `gap_warning`

design look **worthless** on certain embedders. I did not see it coming.

Part 5 is the finding I keep leading with: I shipped a sensible-looking abstention threshold, switched embedders, and watched **every query that should have been refused** sail through as a confident answer. Why a hard-coded similarity threshold is a landmine — and what to do instead.

(The harness itself has kept growing since this was drafted — it now also scores declared-supersession versus timestamps, and a held-out "near-miss" challenge set that no threshold can catch by construction. Both came out of reader comments, and both are covered in [the follow-up post](https://dev.to/gde03/retrieval-augmented-self-recall-what-the-comments-taught-me-re-call-v03-42c1).)

*Part 4 of Retrieval-Augmented Self-Recall. Code: RE-call. The eval-first discipline here is the same one behind Claude Code, Beyond the Prompt.*