Part 4 of Retrieval-Augmented Self-Recall. Code: RE-call. Part 3: the honesty guards. A standard RAG benchmark would have handed my retriever a perfect score on a day it was confidently answering questions it had no data for.
That's not a bug in my system. It's a gap in what those benchmarks ask. Every one of them asks the same thing: when there was an answer, did you rank it first? None of them ask the question that decides whether agent memory is safe: when there was no answer, did you say so?
So RE-call ships its own harness. Here's how it works, and the first finding it produced.
The evaluation runs on 14 answerable queries + 5 unanswerable queries over a synthetic corpus.
Those 5 unanswerable queries are the whole reason the harness exists. They're questions the corpus genuinely cannot answer, where the correct behavior is to abstain — to fire gap_warning
, not to confidently return the nearest memo. Standard retrieval benchmarks are built entirely from answerable queries; they have no way to score "did it correctly say nothing?" This harness is built around that case.
Because there are two jobs — rank well when there's an answer, abstain when there isn't — there are two families of metrics:
**Ranking quality** (for the answerable queries):
**Guard quality** (for the unanswerable queries):
Why you need both is the crux: a system can post excellent MRR and terrible FCR. It ranks beautifully whenever an answer exists, and lies confidently whenever one doesn't. If you only look at ranking metrics — as most RAG evals do — that failure is completely invisible. FCR is what drags it into the light.
The harness runs the full matrix: each embedder (HashingEmbedder
, bge-small
, voyage-3
) crossed with each fusion configuration (dense only → hybrid → hybrid + rerank). That's what lets you answer "which component actually earns its cost?" instead of cargo-culting a reranker into every pipeline.
And it runs against the real thing: 49 integration tests on a live pgvector container (of a 150-test suite), in CI, no mock database. The benchmark exercises the actual retrieval path, not a stand-in.
Here's the ablation on the weak (hashing
) embedder — quality climbs monotonically as you add stages:
| Configuration | MRR | nDCG@10 |
|---|---|---|
| Dense only | 0.63 | 0.72 |
| + sparse (hybrid) | 0.74 | 0.80 |
| + cross-encoder rerank | 1.00 | |
| 1.00 |
Now the same pipeline on the strong bge-small
embedder: dense retrieval already achieves a perfect nDCG@10. Hybrid fusion and reranking add nothing — there's no headroom left to recover.
The conclusion, stated plainly in the writeup: hybrid + rerank buys the most on weaker embedders or harder corpora; on an easy corpus with a strong embedder it's redundant.
That is a genuinely useful engineering result, because the reflex in RAG is to stack a reranker onto everything. This says: don't pay for stages your embedder has already made unnecessary. Measure first. A cross-encoder rerank on every query is real latency and real cost — and on a strong embedder over a well-covered corpus, you're buying zero.
None of these numbers come from an in-memory toy. They come from the same harness that runs in CI against real Postgres — with a dependency audit — every commit. The reason to trust the findings is that the measurement is reproducible. That's the whole pitch of this track: measure honestly, including the parts that make your work look less impressive.
Which is a good segue, because the same ablation surfaced something that made a chunk of my gap_warning
design look worthless on certain embedders. I did not see it coming.
Part 5 is the finding I keep leading with: I shipped a sensible-looking abstention threshold, switched embedders, and watched every query that should have been refused sail through as a confident answer. Why a hard-coded similarity threshold is a landmine — and what to do instead.
(The harness itself has kept growing since this was drafted — it now also scores declared-supersession versus timestamps, and a held-out "near-miss" challenge set that no threshold can catch by construction. Both came out of reader comments, and both are covered in the follow-up post.)
Part 4 of Retrieval-Augmented Self-Recall. Code: RE-call. The eval-first discipline here is the same one behind Claude Code, Beyond the Prompt.