Part 1 of The Answerability Problem. A follow-on from Retrieval-Augmented Self-Recall. That series argued agent memory needs calibrated abstention. This one is what happened when I actually tried to measure it. Code and every number: RE-call (MIT).
There's a benchmark war going on in AI memory, and it's a healthy one.
Mem0's paper (arXiv:2504.19413, Table 2) reports an LLM-as-judge score of J=66.88 for itself against 65.99 for a competitor, Zep. Zep published a rebuttal disputing the methodology and the scores attributed to it. Mem0's later announcement of a new algorithm puts it at 92.5 on LOCOMO and 94.4 on LongMemEval. People argued about all of it. That's how you know the numbers matter to someone.
But both sides are fighting over the same axis: of the questions the memory can answer, how many does it get right?
LOCOMO has five question categories. Four of them are that axis. The fifth is 446 questions, 22.5% of the benchmark, that look answerable and aren't. They name a real person and a real topic from the conversation, then ask about something that person never said. Right event, wrong speaker. The correct answer is "that isn't in here."
I wanted to know how the incumbents score on those 446. So I went and read the harness.
Mem0's published benchmark harness (mem0ai/memory-benchmarks
) contains this:
CATEGORIES_TO_EVALUATE = [1, 2, 3, 4]
Category 5, all 446 adversarial questions, is excluded from scoring entirely.
And the answer prompt contains this instruction:
NEVER say "not specified" … COMMIT AND ANSWER
So it isn't that abstention goes unmeasured. The harness removes the questions that would test it, and then instructs the model not to do it on the ones that remain.
I want to be careful here, because there's a cheap version of this observation and I'm not making it. Excluding cat5 is a defensible choice if what you're measuring is answer quality: a refusal scores zero on an answerable question either way, and mixing the classes muddies the metric. The "COMMIT AND ANSWER" instruction reduces variance from a model that hedges. Neither line is a scandal.
What they are, together, is a scope. The number that comes out the other end is answer accuracy on questions that have answers, and it is structurally incapable of telling you anything about what the system does when the answer isn't there. That's fine, as long as everyone reads it that way. In practice it gets read as "how good is this memory."
Once I was in the harness I kept reading, and the abstention exclusion turns out to be the fourth-most interesting thing in it. The judge:
An independent audit (dial481/locomo-audit
) measured that judge over-accepting 62.8% of deliberately wrong answers. The same audit found LOCOMO's answer key is partly corrupted, which caps the honest achievable ceiling somewhere near 93.6.
Sit with that for a second. A published 92.5 is not "nearly perfect." It is at the ceiling of a benchmark whose ceiling is a data-quality artifact, produced by a judge that accepts nearly two thirds of answers known to be wrong, on the 77.5% of questions that were kept.
I am not claiming anyone cheated. I'm claiming the number is load-bearing for a conclusion it cannot support, and that this is true of every published figure on this benchmark, including any I might publish.
Here's the part that took me longest, and it's the part I'd most like pushback on.
The obvious metric is "how often does it correctly refuse?", which I will call adversarial-abstain. It is completely useless on its own, because I can score 1.00 on it in one line: refuse everything. A memory that always says "I don't know" is perfectly honest and perfectly worthless.
The mirror metric is false-abstain: how often it refuses a question it could have answered. Also gameable on its own, in the opposite direction: never abstain and you score a perfect 0.000. Which, as it happens, is exactly what "COMMIT AND ANSWER" instructs.
So neither column means anything alone. The quantity that can't be gamed by a degenerate policy is the difference:
discrimination = adversarial-abstain − false-abstain
Refuse everything: 1.00 − 1.00 = 0. Refuse nothing: 0.00 − 0.00 = 0. You only score above zero by telling the two classes apart, which is the actual capability in question. It's the same shape as a balanced-accuracy or Youden's J, and I make no claim to having invented it; the claim is that nobody in this space is reporting it, and it is cheap to report.
So I built the harness. It's a couple of hundred lines, it runs on the public LOCOMO file, and it ships in the repo:
curl -sLO https://raw.githubusercontent.com/snap-research/locomo/main/data/locomo10.json
python -m recall.eval.locomo_abstention --data locomo10.json --answerable-sample 40
Then I ran it against my own system.
RE-call, default configuration, on all 446 adversarial questions:
| Mode | Adversarial abstain ↑ | Answerable false-abstain ↓ | discrimination |
|---|---|---|---|
| default | 0.000 [0.00, 0.01] | 0.000 [0.00, 0.01] | 0.000 |
Zero out of 446. Not "worse than I hoped". It is the exact score of a system with no abstention mechanism at all, achieved by a system whose entire published thesis is calibrated abstention. I have spent months writing about this. I have a trust layer, a calibrated cosine threshold, and an entailment judge, and on the hardest public instance of the problem they collectively did nothing.
I'd planned a different article. The draft that sat in this folder for two weeks was called "The one question every AI-memory benchmark skips" and its argument was: they don't measure this, I do, here's my score. I killed it, because the honest version of the finding is not that I found a gap in the field. It's that I found a gap in the field and then fell straight into it.
That is a much more interesting result, and it's what the rest of the series is about.
Before going further, the short version of what I tried. Four modes, each measured against its cost to answerable questions, because a system that abstains on everything scores 1.00 and is useless:
| Mode | Adversarial abstain ↑ | Answerable false-abstain ↓ | discrimination |
|---|---|---|---|
| default | 0.000 | 0.000 | 0.000 |
| calibrated (in-sample, an upper bound) | |||
| 0.574 | 0.420 | 0.154 | |
| entailment judge | 0.347 | 0.290 | 0.057 |
| both | 0.796 | 0.603 | |
| 0.193 |
The best absolute catch is the worst trade: it refuses 60.3% of the questions your users actually wanted answered. There is no row here anyone ships.
And when I later fixed two real retrieval bugs, discrimination went from 0.157 to 0.154, which is unchanged. Both columns rose together. A retrieval improvement cannot improve an answerability judgement, because a better-retrieved on-topic-but-wrong passage scores higher, not lower.
The last time I published a design in this space, the comments were better than the post. Not vague encouragement, but people naming the failure mode precisely and describing what would fix it. I turned those into experiments and shipped what survived. One of them produced a real improvement that Part 2 then shows landing below plain cosine on this workload, which is exactly why it was worth publishing: a design you keep in a drawer accumulates conviction, a design you publish accumulates objections, and objections are the cheapest high-quality signal available to anyone building this stuff.
So if you build or evaluate retrieval-backed agents, the comment section is the point of this, not a formality. Tell me where the measurement is wrong.
One thing worth saying because it cuts against my own framing: the field is already moving. Mem0's newer BEAM benchmark ships an abstention category. Whatever you make of the rest of it, someone over there reached the same conclusion I did and did something about it. I'd rather be early to a fixed problem than right about a permanent one.
If you build or evaluate retrieval-backed agents, I'd genuinely like to know:
Do you measure what your system does when the answer isn't in the corpus, and if you do, what number do you report?
Not "does it hallucinate less." A number, on a fixed set of questions you know are unanswerable, alongside the cost in refused-but-answerable questions. If you have one, I want to see it. If you tried and it came out like mine, that's more useful still.
And if you think discrimination
is the wrong summary statistic, say so. I would rather find that out in a comment section than after building four more articles on top of it.
*Every figure here is measured and reproducible: ***
results/RESULTS.md
§7b and results/FINDINGS.md
§9b. The harness is in the repo. Next: **Part 2, Relevance is not answerability: six signals, and none of them beat plain cosine.