# Retrieval-Augmented Self-Recall — Part 5: The Gap Threshold That Didn't Transfer

> Source: <https://dev.to/gde03/retrieval-augmented-self-recall-part-5-the-gap-threshold-that-didnt-transfer-86a>
> Published: 2026-07-18 12:06:53+00:00

*Part 5 of Retrieval-Augmented Self-Recall. Code: RE-call. Part 4: the eval harness.*

I shipped the `gap_warning`

guard from Part 3 with a sensible-looking default: if the best cosine similarity is **below 0.50**, call it a probable gap and abstain. I tested it. It worked.

Then I swapped the embedding model. **Every question that should have been refused came back as a confident answer.**

No error. No crash. No failed test. The guard was still there, still running, still reporting that everything was fine. It had just quietly stopped being a guard.

Here's the mechanism, because it generalises well past my project, and there's a decent chance it's live in your RAG system right now.

The gap guard fires when `best_cosine < threshold`

. I used 0.50. The problem is that **0.50 means completely different things to different embedders**, because each model lays out its vector space with its own geometry. Same number, different meaning.

The harness measured the cosine distributions for answerable vs. unanswerable queries, per embedder. Look at what "similarity" actually ranges over:

| Embedder | Answerable cosine | Unanswerable cosine | Separable? | FCR @ 0.50 | FCR @ calibrated |
|---|---|---|---|---|---|
`hashing-64` |
0.30 – 0.68 | 0.35 – 0.53 | no — overlap | 0.20* | — |
`bge-small` |
0.70 – 0.90 | 0.51 – 0.64 | yes, at ~0.70 | 1.00 |
0.00 |
`voyage-3` |
0.53 – 0.70 | 0.09 – 0.32 | yes, at ~0.50 | 0.00 |
0.00 |

* *misleadingly low: with overlapping distributions the 0.50 cut also wrongly flags answerable queries, and the error-minimizing threshold simply stops firing at all. No threshold works here.*

Read across the rows and the whole story is there.

** voyage-3:** unanswerable queries score 0.09–0.32, answerable score 0.53–0.70. A threshold of 0.50 lands cleanly in the gap between them. FCR is 0.00. My default worked —

** bge-small:** unanswerable queries score

** hashing-64:** answerable (0.30–0.68) and unanswerable (0.35–0.53)

A magic constant that "works" is the most dangerous kind of code, because it works right up until the context shifts — and then it fails **silently**, which is the worst possible failure mode for a safety guard. My 0.50 wasn't a good threshold that I'd validated. It was a coincidence that held for one embedder and collapsed the moment I changed one.

The fix is cheap and non-negotiable: **calibrate the abstention threshold per embedding model, against a small labeled set.** Twenty-odd queries — a handful answerable, a handful not — is enough to see where the two distributions actually sit and cut between them. Do NOT ship a constant. The writeup says it in one line: *calibrate per embedding model against a small labeled set; do not ship a hard-coded constant.*

Any decision that thresholds a similarity score inherits this exact trap:

In every one of these, the threshold is a property of the **(embedder, corpus) pair**, not a universal constant. Swap the model and your carefully-chosen number is now cutting in the wrong place — and unless you're measuring the failure explicitly, you won't know.

Which is the meta-point, and the reason Part 4 mattered: **this failure is invisible without the eval harness.** The system ranks well, returns plausible results, and lies on gaps. You only catch it by measuring a false-confident rate *per embedder*. Ranking metrics would have shown me green the entire time.

Two footnotes from after this was drafted. First: when I trailed this finding in Part 1, a commenter guessed the mechanism before the post existed — and proposed a *relative* threshold (top hit versus the rest of the batch) instead of an absolute one. That experiment is still owed; my worry is that a spread-based check is blind to the single confident distractor. Second, and worse: there is a whole failure class **no threshold can catch, by construction** — the near-miss that scores *high*. That one needed a different kind of fix, and it's the subject of [the follow-up post](https://dev.to/gde03/retrieval-augmented-self-recall-what-the-comments-taught-me-re-call-v03-42c1).

If a better *threshold* helps, would a better *embedding* help more? The intuitive next move is to fine-tune the embedder on my domain. I did. The result was **zero** — and Part 6 is about why that was exactly the right outcome, and the one case where fine-tuning actually did move the needle.

*Part 5 of Retrieval-Augmented Self-Recall. Code: RE-call. The "measure the failure that matters, not the one that flatters" discipline runs through Claude Code, Beyond the Prompt too.*