Retrieval-Augmented Self-Recall — Part 3: Teaching RAG to Say \"I Don't Know\

A developer building RE-call, a retrieval system for agent memory, introduces honesty guards to prevent agents from acting on hallucinated or irrelevant results. The gap_warning flag checks if the top retrieval score meets a calibrated threshold, while freshness warns when retrieved content is stale. The post highlights that good ranking can worsen failures by returning confident noise, and that honesty guards are critical for trustworthy agent decisions.

Part 3 of Retrieval-Augmented Self-Recall. Code: RE-call. Part 2: hybrid retrieval on Postgres. Ask your agent "have we tried this filter on this market before?" when the honest answer is never . A ranking retriever hands back the three closest memos anyway — something about a different filter, on a different market — and the agent, looking at three confident results, concludes: yes, we've looked at this. It just made a decision on a hallucination. Nothing in the stack noticed. No error was raised, because from the retriever's point of view nothing went wrong: you asked for the nearest neighbours and it gave you the nearest neighbours. Everything in Part 2 made retrieval good . Good ranking makes this failure worse , not better — it returns confident noise faster. This post is about making retrieval honest , which for agent memory is the part that actually decides whether you can trust it. So RE-call wraps retrieval in honesty guards — this post covers the original three, each answering a question ranking metrics never ask. The current repo has grown that table to six, and the growth story is its own post https://dev.to/gde03/retrieval-augmented-self-recall-what-the-comments-taught-me-re-call-v03-42c1 : two of the new guards exist because readers of Part 1 pointed at exactly the weaknesses you're about to see me describe. I'll flag those spots as we go. gap warning — "is the best match good enough to trust?" After retrieval, look at the best dense cosine similarity. If it falls below a calibrated threshold , the top result isn't the answer — it's the least-bad noise. The system sets gap warning = true . The important design idea: this is a second-order signal . Retrieval still returns its ranked list; the guard annotates how much to trust it. That annotation is what lets the calling agent do something other than blindly act — it can abstain, ask a human to confirm, widen the search, or explicitly note "no prior memory on this" before proceeding. That single flag is the difference between an agent that says "we've looked at this before" and one that says "I have nothing relevant on this — treat it as new." In a system that makes decisions, that distinction is worth more than any ranking improvement. There's one buried landmine here: what threshold? The obvious move is to pick something like 0.50 and move on. That obvious move is quietly, dangerously wrong — and it's the biggest finding in this series, so I'm giving it its own post Part 5 . For now, the load-bearing word is calibrated : the threshold is fit to data, never hard-coded. Every memo has a timestamp. The freshness guard reports the age of retrieved content and warns when it's stale relative to the re-index cadence my corpus re-indexes daily, so "stale" has a concrete meaning . This one is specific to memory in a way document QA rarely deals with. A documentation corpus is mostly static — last year's page is still roughly true. Agent memory is a moving target: a decision recorded in April may have been reversed in June. Without a freshness signal, April-truth and June-truth are indistinguishable at retrieval time, and the agent will happily act on a superseded conclusion. Freshness lets it weight recency, or at least flag the risk. Honest update, because this section aged: freshness turned out to be the weakest guard of the three, and a commenter on Part 1 put a finger on why — supersession is a relation between two memos , and no per-document timestamp can see a relation. We later measured it: even a steelmanned "trust the newest relevant hit" heuristic still hands back the stale memory 83–100% of the time , while an explicitly declared supersedes: link holds at 0.00 . The fix a trust layer that binds the relation at write time and the measurement are in the follow-up post https://dev.to/gde03/retrieval-augmented-self-recall-what-the-comments-taught-me-re-call-v03-42c1 . The most agent-specific guard of the three. Before the agent proposes an idea, it queries memory for closed decisions on that topic — the "we tried X, it failed, here's why" memos — and the guard surfaces them. The failure it prevents is subtle and expensive: an agent re-proposing a dead idea because the memo that killed it three months ago didn't happen to rank in the top results for today's phrasing. Ranking-optimized retrieval is bad at this specifically, because a settled-decision memo is often lexically distant from the fresh proposal even though it's the most decision-relevant document in the store. The implementation leans on structure: decision-type memos closed hypotheses, postmortems are typed, and a targeted retrieval path prioritizes them when the agent is in "propose" mode. Memory that can't defend its own past decisions is condemned to relive them. Retrieval answers one question: what's closest? The guards answer the three that actually govern whether the agent should act: gap warning That's the whole difference between a search index and a memory . A search index ranks. A memory knows its own limits. Since this was drafted, the guard table grew: trust verdicts with declared supersession, an opt-in entailment judge for the high-similarity-but-wrong case a threshold can never catch, and a write-time lint for the supersession graph. All three exist because readers argued with this post's ancestors — that story, with measurements, is the follow-up https://dev.to/gde03/retrieval-augmented-self-recall-what-the-comments-taught-me-re-call-v03-42c1 . A guard only helps if its signal reaches the decision layer . A gap warning that gets computed and then dropped before the agent sees it is worse than useless — it's false assurance that the system is careful when it isn't. So in RE-call the honesty signals ride inside the retrieval result: you cannot get the answer without also getting "here's how much to trust it." If you read the applied series, this is the same principle as making the tool enforce the rule instead of trusting the prompt to. The guards make claims: this is a gap, this is stale. Claims demand measurement — and "how well does it know when it doesn't know?" is a metric that standard RAG benchmarks don't even have. Part 4 builds the eval harness that measures it, and delivers the first finding about which pipeline components actually earn their cost. Part 3 of Retrieval-Augmented Self-Recall. Code: RE-call. This is the layer that makes Claude Code, Beyond the Prompt's memory trustworthy, not just searchable.