AI
AI abstention separates trustworthy AI memory from confident fabrication. Why a real 'no evidence found' response is harder to engineer than retrieval.
A clinician asks a hospital AI assistant for a patient’s latest kidney panel. No such labs exist - they were never ordered - so the assistant retrieves the closest thing it can find and fluently summarizes a different patient’s results. The scenario is hypothetical; the mechanics are not, and they come down to a property the field calls AI abstention: whether a system can say “no evidence found” instead of manufacturing an answer.
We now know why systems fail at this, and the answer is uncomfortable. In September 2025, OpenAI published research on why language models hallucinate, an argument that has since appeared in Nature as “Evaluating large language models for accuracy incentivizes hallucinations”: the field has been training models to guess. Benchmarks score accuracy, accuracy rewards a confident answer on every question, and “I don’t know” scores zero - the same as a wrong answer - so guessing is the dominant strategy and models learn it thoroughly. The paper’s SimpleQA comparison makes the point with numbers. OpenAI’s older o4-mini abstained on 1% of questions, answered 24% correctly, and got 75% wrong. The newer gpt-5-thinking-mini abstained on 52%, answered 22% correctly, and got 26% wrong. Accuracy differs by two points. The error rate differs by a factor of three. The entire difference is the willingness to say nothing.
This article takes that argument to the layer where it bites hardest: memory. I covered why AI agents forget by design - statelessness is the architecture every provider chose, and everything we call memory is scaffolding built around it. That scaffolding is where abstention gets engineered or doesn’t, and the industry’s fast-growing catalog of memory layers is benchmarked, marketed, and bought almost entirely on what those layers get right. A model that guesses is annoying. A memory system that fabricates is worse, because memory is the component everything downstream treats as ground truth. It cannot be institutional memory if it invents institutions.
What AI abstention means in a memory system# #
The research community has treated abstention as a serious problem for a while - the TACL survey “Know Your Limits” catalogs a whole field of work on when models should decline to answer - but almost all of that work targets the model. In a memory system, abstention - the system’s ability to respond “no relevant evidence” as an explicit, deliberate output rather than returning the closest available text - is mostly not a model property. It is a pipeline property, and a perfectly calibrated model cannot save a memory pipeline that feeds it garbage with a straight face.
A second distinction keeps the rest of this article honest. Refusal is a policy behavior: the system could answer but declines for safety or scope reasons. Abstention is epistemic: the system recognizes that its evidence does not support an answer. A memory layer needs both, but abstention is the one that determines whether you can trust what it does return. Every retrieval that comes back with content is implicitly claiming “this is what we know.” If the system physically cannot say “we know nothing relevant,” that claim is unfalsifiable from the inside - and the burden of catching fabrication lands on the user, the one person with the least visibility into what the store actually contains.
That is why I’d rank abstention above headline accuracy when evaluating memory infrastructure, and the business version of the argument is short. An assistant that says “I don’t know, can you point me to it?” costs you a follow-up question. An assistant that confidently invents an answer costs you whatever decision got made on top of it. Accuracy measures how often the system is right when it answers. Abstention determines whether “it answered” carries any information at all.
Four reasons a memory system should abstain# #
“I don’t know” sounds like one behavior. In a production memory system it is at least four distinct states, and they demand different downstream behavior.
Nothing stored. The genuine absence case: the user asks about a deployment nobody ever discussed with the system. The honest answer is “no memory exists on this,” ideally with an invitation to supply the missing context.
Below threshold. Evidence was retrieved, but nothing clears a defensible confidence bar. Three fragments at similarity 0.41 to 0.48 against a 0.6 threshold are not an answer; they are noise wearing an answer’s clothes. The correct behavior is to abstain, and optionally to surface the fragments as explicitly low-confidence leads rather than as a synthesized reply.
Not permitted. The evidence exists, but the caller lacks access to it. This case has a trap inside it: the abstention must not leak existence. “There are three documents about the reorg but you can’t see them” is an access-control failure dressed up as good behavior. The response has to be indistinguishable, from the caller’s side, from the nothing-stored case.
Was erased. The evidence existed and was deliberately deleted - a retention policy, a privacy request, an explicit correction. Same trap, sharper edge: confirming that something was erased confirms that it existed, which is often exactly what the erasure was meant to prevent. And the system itself must not resurrect the deleted fact from some secondary index that didn’t get the memo - a risk that is measured, not hypothetical: a June 2026 study of ghost vectors showed that soft-deleted embeddings remain reconstructible inside HNSW vector indexes, because deletion at the metadata level leaves the vector itself recoverable underneath.
The taxonomy matters because conflating the cases produces trust failures that look like unrelated bugs. Treat “below threshold” like “nothing stored” and the system discards weak-but-real signals a human could have validated. Treat “not permitted” like “below threshold” and you leak existence through the shape of the response. Treat “was erased” like “nothing stored” internally and a stale replica quietly undoes a legal deletion. A trustworthy memory layer distinguishes all four internally, even when two of them must present identically to the outside.
Why retrieval pipelines guess instead# #
The short answer: because they are built on retrieval primitives that cannot say no, wrapped in pipelines that are graded on recall.
Nearest-neighbor search returns the nearest neighbors. That is the whole contract. Pinecone’s query API returns the top_k
closest vectors ordered by similarity; pgvector answers ORDER BY embedding <=> query LIMIT k
and leaves any notion of “close enough” to a WHERE clause you write yourself. Cosine similarity always produces a ranking, even when the best match is garbage. There is no native no-match state anywhere in the primitive - which is a large part of why pure similarity search makes abstention hard, and it deserves its own treatment.
Watch what the naive pipeline does with a question it has no basis to answer:
from sentence_transformers import SentenceTransformer
import numpy as np
model = SentenceTransformer("all-MiniLM-L6-v2")
corpus = [
"Customer ACME-48201 reported a login loop after the SSO migration on May 12.",
"Refund of 240 EUR issued to ACME-48201 for double-billed April invoice.",
"ACME-48201 is on the Team plan, renewal date September 1, 30 seats.",
]
corpus_vecs = model.encode(corpus, normalize_embeddings=True)
query = "What did ACME-48201 say about the data export failing?"
query_vec = model.encode([query], normalize_embeddings=True)[0]
scores = corpus_vecs @ query_vec # cosine similarity via dot product
best = int(np.argmax(scores))
print(f"Retrieved (score {scores[best]:.2f}): {corpus[best]}")
Nothing about data export exists in the store, and the pipeline still returns the login incident at a similarity of 0.45 - a number that reads like moderate relevance and carries no warning at all. (The scores are real; I ran this exact snippet while writing.) Feed that context to an LLM with the standard “answer using the context above” prompt and you get a fluent response about the export problem, grounded in a memory about something else entirely. The failure has no floor, either: swap the encoder for plain TF-IDF and ask something with zero lexical overlap, and argmax
still returns a document when every score is 0.000. The user sees confidence. The system experienced a near-miss it had no vocabulary to report.
The naive fix is a threshold: if the top score is below some cutoff, return nothing. Directionally correct, and in practice this is where the trouble starts. Raw similarity is not calibrated - 0.6 means different things across embedding models, corpus domains, even query lengths - so teams tune the cutoff on a handful of examples, watch it reject queries it should have answered, and respond the way recall-graded teams always respond: they lower it, or raise k. Every lowered threshold converts a visible annoyance (the system said it didn’t know) into an invisible one (the system answered from irrelevant evidence). The failure mode of an aggressive threshold is a support ticket. The failure mode of a permissive one is a wrong decision nobody traces back for months.
The other tempting remedy has a measured hole in it. Sampling the pipeline several times and abstaining when the answers disagree assumes errors are unstable - and a June 2026 study of retrieval-state lock-in found that at five samples per question, 42% of errors in knowledge-graph RAG and 59% in dense-retrieval RAG showed zero answer dispersion. The pipeline retrieves the same wrong evidence every time and converges, unanimously, on the same wrong answer. Consensus confidence cannot catch an error the retriever itself keeps making.
And waiting for better models is not a plan either. AbstentionBench, Meta FAIR’s NeurIPS 2025 evaluation of 20 frontier LLMs across 20 datasets, found abstention essentially unsolved, barely improved by scale - and actively degraded by reasoning fine-tuning, by 24% on average, even in the math and science domains the training targeted. So the fix is not a better LLM and not more sampling. It is a retrieval pipeline that knows when its own evidence is insufficient: calibration, thresholds, and a response type the caller is forced to handle.
The benchmark evidence# #
The field’s own benchmarks already treat abstention as a measured category. What they show is a gap between what gets marketed and what gets tested.
LongMemEval, the ICLR 2025 benchmark that has become the standard yardstick for long-term conversational memory, contains 500 questions across five abilities: information extraction, multi-session reasoning, temporal reasoning, knowledge updates, and abstention. The abstention subset is 30 questions whose defining property is that the answer never appeared in the interaction history; the only correct behavior is to decline. The arithmetic sets the stakes: a system that always abstains scores exactly 6% - the abstention slice and nothing else - and a system that never abstains caps out at 94%. Every headline score above roughly 90% is therefore making an implicit claim about abstention behavior that almost nobody checks.
Now the uncomfortable part: try to find that kind of breakdown for the memory products you are evaluating. Vendors publish LongMemEval headline numbers enthusiastically. Abstention sub-scores, almost never. The disclosures I could actually locate as of July 2026:
| System | Headline score (variant, as reported) | Abstention sub-score |
|---|---|---|
OpenDBMemoriaThree caveats before anyone puts these in a slide. The runs use different dataset variants, reader models, and judges, so the rows are not comparable to each other - only the gap within a row is informative. Second, LongMemEval’s history sessions were cleaned up in September 2025, so scores published before and after that point are not directly comparable either; always ask which version a number was produced on. Third, Memoria’s own breakdown shows why the sub-score matters: the same memory store swings from 56.7% to 93.3% on abstention purely by changing the reader model, while the headline moves far less. The abstention column is where the variance hides.
The pattern extends beyond one benchmark. HaluMem, the first benchmark to instrument memory systems operation by operation, traced where memory hallucinations originate and found them accumulating in the extraction and update stages before propagating into answers - the fabrication is often baked into the store before any question arrives. Stanford’s 2026 AI Index reports hallucination rates from 22% to 94% across 26 top models when a false premise is framed as the user’s own belief; the systems most eager to please are the least able to say no, and a memory layer is, by construction, the most user-adjacent component in the stack. And the newest benchmark generation is starting to encode all this properly: LongMemEval-V2, released in May 2026 for agent-trajectory memory, scores premise awareness - whether the system notices that the question itself assumes something false. Abstention is graduating from a 30-question afterthought to a category of its own. The gap between a product’s headline score and its abstention sub-score is, in my read, the single best public predictor of how it will behave in front of your users.
What good abstention looks like in production# #
The engineering is unglamorous, which is probably why nobody markets it. Five components, in dependency order.
Calibrated confidence at retrieval. Raw cosine similarity is not a probability. A 0.72 means different things in different embedding spaces, corpora, and query distributions. Calibration - mapping raw scores to something that empirically tracks correctness on held-out data from your actual corpus - is the foundation everything else stands on. Skip it and every threshold downstream is a guess dressed as a number.
Confidence that survives composition. Real answers get assembled from several memories, and here is where implementations quietly cheat: three memories at confidence 0.6, 0.7, and 0.5 do not compose into a 0.7-confidence answer, which is what max-pooling gives you. If the answer depends on all three being right, the honest number sits closer to their product than their max. Any system whose answer confidence equals its best fragment’s confidence is inflating.
A threshold someone can defend. There will still be a cutoff, and the difference between a defensible one and a guess is whether it was set against measured calibration curves and whether it is configurable per deployment - a brainstorming assistant and a compliance assistant should not share a tolerance for being wrong. There is also good evidence one threshold is not enough: a July 2026 analysis of the two axes of abstention shows that a single confidence cutoff cannot separate “the model would be wrong” from “the question has no answer,” and argues for scoring answerability separately. That maps exactly onto the four-reasons taxonomy above: nothing-stored is an answerability failure, below-threshold is a correctness one.
A structurally distinct response type. This is the one I’d weight highest, and the one I see least - it separates engineering from theater. If abstention arrives as a friendly sentence inside a normal-looking answer payload, the caller’s code cannot branch on it, the UI renders it like an answer, and downstream automations happily consume it as one. Abstention has to be a different shape:
// The response contract for a memory query. Abstention is not a sad
// string inside an answer - it is a different variant the caller
// must explicitly handle, with the cause distinguished.
type MemoryResponse =
| {
status: "found";
evidence: Memory[];
composite_confidence: number; // calibrated, propagated - not max-pooled
}
| {
status: "no_match";
cause: "nothing_stored" | "below_threshold" | "not_permitted" | "was_erased";
leads?: Memory[]; // optional low-confidence fragments, labeled as such
};
A type checker now enforces what a prompt never could: every caller decides what its UI does when memory comes back empty, per cause. Note the deliberate asymmetry - not_permitted
and was_erased
exist as internal causes, but a correctly built API collapses them to look like nothing_stored
at any trust boundary where existence itself is sensitive.
Reason codes, logged. Which is the fifth component: the four causes must be recorded even where they cannot be revealed, because audit is where governance abstentions prove they happened for the right reason. Policy you cannot express in the response shape is policy you do not actually have.
Almost nobody ships the full set today. One vendor gets partial credit: OMEGA documents an abstention floor in its retrieval architecture - results below fixed relevance thresholds are discarded rather than surfaced - while its published 95.4% LongMemEval result reports no abstention sub-score. That is roughly the state of the art in disclosure: the mechanism exists in the docs, the measurement isn’t in the marketing. None of this is a research program. It is a few weeks of deliberate engineering - a calibration harness, composition math, threshold configuration, a response union type, audit logging - that compounds into months of user trust. The asymmetry is the point: the cost is bounded, and the failure it prevents is not.
The regulatory case, briefly# #
Confident-wrong already has a legal price tag. In Moffatt v. Air Canada, the British Columbia tribunal held the airline liable after its website chatbot invented a bereavement refund policy, rejecting the argument that the chatbot was a separate entity responsible for its own statements. The invented policy was a retrieval-and-abstention failure of exactly the kind above: the correct answer existed on another page, the bot asserted its own version instead, and no part of the stack could say “I am not certain about refund policy.” The damages were small. The precedent - your system’s confident fabrications are your representations - was not. Outside regulated work the stakes look softer, but the team cost of confident wrong answers compounds the same way; it just lands in rework and eroded trust rather than tribunal orders.
The compliance clock is also running. Under the EU AI Act, the Commission’s enforcement powers over general-purpose AI providers activate on August 2, 2026 - twelve days after this piece publishes - with fines for GPAI violations reaching 15 million euros or 3% of worldwide turnover, and high-risk systems must achieve and declare appropriate levels of accuracy and keep automatic logs over their lifetime. A memory layer that cannot distinguish “no evidence” from a fabricated answer has no honest way to make that declaration, and nothing meaningful to log when it fails. In US healthcare, the ONC’s HTI-1 rule already requires certified predictive decision-support tools to expose 31 source attributes covering training data, validation, and performance so clinicians can judge whether an output deserves trust. Clinicians themselves are naming the gap - a May 2026 NEJM perspective asks directly “Can AI Say ‘I Don’t Know’?” and observes that clinicians are expected to disclose the limits of their knowledge while the AI tools entering their workflow often cannot. This is the same regulatory clock driving the portability question - the statutes treat memory as consequential infrastructure, whether or not vendors do.
How to evaluate abstention when you’re picking a memory product# #
Benchmarks aside, abstention is unusually easy to probe in a sales demo, because the vendor cannot prepare for questions about things that were never stored. Five questions, and what a good answer looks like:
| Ask the vendor | What good looks like |
|---|---|
| Demo a query whose correct answer is “no information available” | An explicit, structured no-match response - not a summary of the nearest unrelated memory, not a fluent guess |
| Show your LongMemEval abstention sub-score, not the headline | A number, with the dataset version and judge named. “We don’t break that out” is itself an answer |
| Is abstention a distinct response type or different text? | A schema-level variant the calling code must handle, with cause codes - not an apology string in a normal payload |
| How is confidence calibrated, and where does the threshold live? | Calibration against held-out data, a configurable threshold per deployment, and an explanation of how confidence composes across multiple memories |
| How do you distinguish nothing-stored, low-confidence, permission-blocked, and erased? | Four internal causes, logged for audit, with the two governance cases indistinguishable from absence at the API boundary |
Most vendors will struggle with most of these. That is not cynicism; it is the current state of a market that has been optimizing headline recall because headline recall is what buyers ask about. The questions score more than today’s product - they score whether the team building it believes silence is a feature. Change what you ask about and the market will follow; it did for uptime, and it did for SOC 2.
So what?# #
When you next evaluate an AI memory product, ask for a live demo of a query whose correct answer is “no relevant memory found.” If it fabricates, summarizes the closest unrelated thing, or returns a low-confidence answer with no confidence signal, you are looking at a product that will quietly manufacture trust failures in production. “I don’t know” has to be a real response the architecture supports - a distinct type with a cause attached, not an apology formatted like an answer. Most aren’t built that way.