{"slug": "retrieval-augmented-self-recall-part-3-teaching-rag-to-say-i-don-t-know", "title": "Retrieval-Augmented Self-Recall — Part 3: Teaching RAG to Say \\\"I Don't Know\\", "summary": "A developer building RE-call, a retrieval system for agent memory, introduces honesty guards to prevent agents from acting on hallucinated or irrelevant results. The gap_warning flag checks if the top retrieval score meets a calibrated threshold, while freshness warns when retrieved content is stale. The post highlights that good ranking can worsen failures by returning confident noise, and that honesty guards are critical for trustworthy agent decisions.", "body_md": "*Part 3 of Retrieval-Augmented Self-Recall. Code: RE-call. Part 2: hybrid retrieval on Postgres.*\n\nAsk your agent *\"have we tried this filter on this market before?\"* when the honest answer is *never*. A ranking retriever hands back the three closest memos anyway — something about a different filter, on a different market — and the agent, looking at three confident results, concludes: yes, we've looked at this.\n\nIt just made a decision on a hallucination. Nothing in the stack noticed. No error was raised, because from the retriever's point of view nothing went wrong: you asked for the nearest neighbours and it gave you the nearest neighbours.\n\nEverything in Part 2 made retrieval *good*. Good ranking makes this failure **worse**, not better — it returns confident noise faster. This post is about making retrieval *honest*, which for agent memory is the part that actually decides whether you can trust it.\n\nSo RE-call wraps retrieval in **honesty guards** — this post covers the original three, each answering a question ranking metrics never ask. (The current repo has grown that table to six, and the growth story is [its own post](https://dev.to/gde03/retrieval-augmented-self-recall-what-the-comments-taught-me-re-call-v03-42c1): two of the new guards exist because readers of Part 1 pointed at exactly the weaknesses you're about to see me describe. I'll flag those spots as we go.)\n\n`gap_warning`\n\n— \"is the best match good enough to trust?\"\nAfter retrieval, look at the best dense cosine similarity. If it falls **below a calibrated threshold**, the top result isn't the answer — it's the least-bad noise. The system sets `gap_warning = true`\n\n.\n\nThe important design idea: this is a **second-order signal**. Retrieval still returns its ranked list; the guard *annotates* how much to trust it. That annotation is what lets the calling agent do something other than blindly act — it can abstain, ask a human to confirm, widen the search, or explicitly note \"no prior memory on this\" before proceeding.\n\nThat single flag is the difference between an agent that says *\"we've looked at this before\"* and one that says *\"I have nothing relevant on this — treat it as new.\"* In a system that makes decisions, that distinction is worth more than any ranking improvement.\n\nThere's one buried landmine here: **what threshold?** The obvious move is to pick something like 0.50 and move on. That obvious move is quietly, dangerously wrong — and it's the biggest finding in this series, so I'm giving it its own post (Part 5). For now, the load-bearing word is *calibrated*: the threshold is fit to data, never hard-coded.\n\nEvery memo has a timestamp. The freshness guard reports the **age** of retrieved content and warns when it's stale relative to the re-index cadence (my corpus re-indexes daily, so \"stale\" has a concrete meaning).\n\nThis one is specific to *memory* in a way document QA rarely deals with. A documentation corpus is mostly static — last year's page is still roughly true. Agent memory is a moving target: a decision recorded in April may have been *reversed* in June. Without a freshness signal, April-truth and June-truth are indistinguishable at retrieval time, and the agent will happily act on a superseded conclusion. Freshness lets it weight recency, or at least flag the risk.\n\n**Honest update, because this section aged:** freshness turned out to be the weakest guard of the three, and a commenter on Part 1 put a finger on why — supersession is a *relation between two memos*, and no per-document timestamp can see a relation. We later measured it: even a steelmanned \"trust the newest relevant hit\" heuristic still hands back the stale memory **83–100% of the time**, while an explicitly declared `supersedes:`\n\nlink holds at **0.00**. The fix (a trust layer that binds the relation at write time) and the measurement are in [the follow-up post](https://dev.to/gde03/retrieval-augmented-self-recall-what-the-comments-taught-me-re-call-v03-42c1).\n\nThe most agent-specific guard of the three. Before the agent proposes an idea, it queries memory for **closed decisions** on that topic — the \"we tried X, it failed, here's why\" memos — and the guard surfaces them.\n\nThe failure it prevents is subtle and expensive: an agent re-proposing a dead idea because the memo that killed it three months ago didn't happen to rank in the top results for today's phrasing. Ranking-optimized retrieval is bad at this specifically, because a settled-decision memo is often *lexically* distant from the fresh proposal even though it's the most decision-relevant document in the store.\n\nThe implementation leans on structure: decision-type memos (closed hypotheses, postmortems) are typed, and a targeted retrieval path prioritizes them when the agent is in \"propose\" mode. Memory that can't defend its own past decisions is condemned to relive them.\n\nRetrieval answers one question: *what's closest?* The guards answer the three that actually govern whether the agent should act:\n\n`gap_warning`\n\n)That's the whole difference between a **search index** and a **memory**. A search index ranks. A memory knows its own limits.\n\n(Since this was drafted, the guard table grew: trust verdicts with declared supersession, an opt-in entailment judge for the high-similarity-but-wrong case a threshold can never catch, and a write-time lint for the supersession graph. All three exist because readers argued with this post's ancestors — that story, with measurements, is [the follow-up](https://dev.to/gde03/retrieval-augmented-self-recall-what-the-comments-taught-me-re-call-v03-42c1).)\n\nA guard only helps if its signal **reaches the decision layer**. A `gap_warning`\n\nthat gets computed and then dropped before the agent sees it is worse than useless — it's false assurance that the system is careful when it isn't. So in RE-call the honesty signals ride *inside* the retrieval result: you cannot get the answer without also getting \"here's how much to trust it.\" (If you read the applied series, this is the same principle as making the *tool* enforce the rule instead of trusting the prompt to.)\n\nThe guards make claims: *this is a gap, this is stale.* Claims demand measurement — and \"how well does it know when it doesn't know?\" is a metric that standard RAG benchmarks don't even have. Part 4 builds the eval harness that measures it, and delivers the first finding about which pipeline components actually earn their cost.\n\n*Part 3 of Retrieval-Augmented Self-Recall. Code: RE-call. This is the layer that makes Claude Code, Beyond the Prompt's memory trustworthy, not just searchable.*", "url": "https://wpnews.pro/news/retrieval-augmented-self-recall-part-3-teaching-rag-to-say-i-don-t-know", "canonical_source": "https://dev.to/gde03/retrieval-augmented-self-recall-part-3-teaching-rag-to-say-i-dont-know-28no", "published_at": "2026-07-18 12:06:01+00:00", "updated_at": "2026-07-18 12:28:50.715754+00:00", "lang": "en", "topics": ["artificial-intelligence", "machine-learning", "ai-agents", "ai-safety", "developer-tools"], "entities": ["RE-call", "Postgres"], "alternates": {"html": "https://wpnews.pro/news/retrieval-augmented-self-recall-part-3-teaching-rag-to-say-i-don-t-know", "markdown": "https://wpnews.pro/news/retrieval-augmented-self-recall-part-3-teaching-rag-to-say-i-don-t-know.md", "text": "https://wpnews.pro/news/retrieval-augmented-self-recall-part-3-teaching-rag-to-say-i-don-t-know.txt", "jsonld": "https://wpnews.pro/news/retrieval-augmented-self-recall-part-3-teaching-rag-to-say-i-don-t-know.jsonld"}}