Part 4 of The Answerability Problem, and the one that isn't about abstention. Parts 1–3 argued that the field measures the wrong half and that my own system hits a wall on the right half. This part is the ledger: what RE-call is, what it loses at, and why you might still want the thing that loses. Code: RE-call (MIT).
Three parts of "here is what doesn't work" earn one part of "here is what this is for."
So, plainly: on BEAM's 1M-token bucket, Mem0 scores better than RE-call on categories I care about. I'm not going to bury that, and I'm not going to explain it away. I'm going to tell you exactly why it happens, what that accuracy costs, and let you decide which side of the trade you want, because for a lot of people the answer is not the obvious one.
Mem0 calls an LLM when you write a memory, and stores the distilled result. RE-call calls no LLM ever, and stores your raw turns.
That single decision explains everything downstream: the wins, the losses, the bill, and where your data goes.
The clearest place to see it is temporal_reasoning
, my second-worst BEAM category: 0.408 against Mem0's 0.567.
One disclosure before that number goes anywhere, because it changes what it means: that cell was measured with reranking off. The BEAM harness takes a --reranker
flag and defaults it to none
, and the run didn't pass it. Reranking is the largest retrieval gain in this project. Part 2 measures it at hit@5 0.671 → 0.777 on LOCOMO, improving every category including the multi-hop floor I'd predicted it wouldn't touch. So 0.408 is my shipped default, not my best configuration, and the reranked cell is unmeasured.
I'm not going to tell you which way that would go. The last time I predicted a category wouldn't benefit from reranking I was wrong, and guessing here would be the same error with a bigger number attached.
What I can say is that reranking is unlikely to be the whole story, because the diagnosis isn't a ranking failure. Of seven badly-lost questions only one had empty retrieval; five were answered confidently and wrongly. They are all the same shape, "how many days between A and B", and in every one my system used the wrong instance of a date:
| gold | our answer |
|---|---|
| 25 Mar → 1 Apr = 7 days | |
| 14 days, using the updated deadline of 15 Apr | |
| 25 Mar → 10 Apr = 16 days | |
| 26 days, using a different viewing on 15 Mar | |
| 15 Feb → 20 Feb = 5 days | |
| 0 days, using 10 Jan, the date the deadline was set | |
Mem0 gets these right because its stored memory is one distilled line, "Sprint 1 deadline: February 15, 2024". Mine is the same date scattered across many raw turns in different roles: when it was set, when it was revised, when someone mentioned it in passing.
This is the one category where LLM distillation at ingest is genuinely the better architecture, and no retrieval-side change I can afford replicates it. It's recorded in the repo as a known limit, not an open task. I tried the obvious fix, preferring the newest instance, and it is wrong: sometimes the correct date is the older one, the original deadline rather than the revision. That kills every recency heuristic, which is the third independent line of evidence against newest-wins in this project.
So: they're better at this, for a real reason, and I can't cheaply fix it.
Here's the other side of the same decision, measured on the identical benchmark workload:
| RE-call | Mem0 | |
|---|---|---|
| LLM calls to build the memory | 0 | |
| 272 | ||
| tokens | 0 | |
| 2.6 M | ||
| cost | ||
| $0 | ||
| $7.29 | ||
| ingest wall clock | 67 s | |
| 288 s (~4.3× slower) | ||
| where your documents go | your Postgres | an LLM provider, once per memory written |
That $7.29 is for one benchmark's memory. It is not a subscription. It is a per-memory marginal cost that scales with everything you ever write. RE-call's write path calls no model, so its marginal cost is $0 at any scale, on any model, forever. There is no pricing change upstream that can alter that number.
The ingest gap is the same fact wearing a stopwatch: an extraction call per session is a network round-trip per session.
(Retrieve latency, 77 ms against 104 ms, I report as directional only. The repeated-query bootstrap CI is optimistic and the two backends differ, so I won't lean on it.)
Your data never leaves your infrastructure. Local embeddings, the PostgreSQL you already run, and it works offline: on an air-gapped box, in a privacy-bound environment, under a DPA that doesn't have room for another subprocessor.
A memory layer that calls an LLM per write cannot offer that, structurally. Not because anyone is careless, but because the architecture requires sending the content out to distil it.
If you're a solo developer, that's a cost story. If you're a company holding customer conversations, it's a procurement story, and it's usually the one that decides the question before accuracy is ever discussed. The cloud embedder is available as a measured option rather than a default, and the repo prices it honestly: it wins on 16 of 17 held-out corpora, median +0.059 hit@5, and it means every document and every query leaves your machine, at 246 ms p50 against 45 ms local. You get the numbers and you choose. That's the pattern for everything here.
This is the part I undersold for five articles.
bge-small
, bge-large
, Voyage, or anything OpenAI-compatible. Measured across 17 corpora, with the rule for when paying is worth it.I'd rather ship a flag with two measured settings than a default with a marketing claim.
BEAM has an abstention
category, questions whose correct answer is "that isn't in here." I scored Mem0's own published answers on it, with Mem0's own judge. n=70:
| Mem0 did | n | mean score |
|---|---|---|
| abstained | 38 | 0.974 |
| answered anyway | 32 | 0.016 |
The category is near-perfectly binary, and it is testing exactly one thing: does the system invent an answer when the evidence isn't there. It invents one 46% of the time.
One real example: asked about user feedback that was never recorded, it answered "User testing showed a positive response: the dynamic language switching feature achieved a 90% satisfaction rate." The corpus does contain "achieving a 90% satisfaction rate is a strong start", the assistant speculating, which retrieval surfaced and the answerer read as fact.
I want to be precise about what I'm claiming, because this is where it would be easy to overreach:
So neither of us solves this. The difference is the shape of the failure: a system that fabricates a plausible satisfaction rate is a different kind of liability from one that returns nothing. Depending on what you're building, "slightly less accurate" and "confidently invents a statistic" are not two points on one scale.
Worth saying because they cut against my own story as much as anyone's.
BEAM's unanswerable questions score higher than its answerable ones: median top-1 cosine
The scoring is 9:1 against withholding. Abstention is ~10% of BEAM, so a policy that abstains more gains on 30 questions and loses on 270. That is not a complaint about the benchmark. Its abstention category is well built. It means an abstention claim cannot be made through BEAM's aggregate, no matter how good the policy gets. The field needs a metric that prices a false answer against a withheld one, and this isn't it.
Two smaller corrections that arm produced: BEAM's harness pins gpt-5 as answerer and judge (not gpt-4o), and the published 64.1 is a mean rubric-nugget score, not a pass rate: the pass rate for the same run is 70.14%.
What I'm not reporting: a paired BEAM aggregate. I have per-category cells and the abstention probe; I do not have a like-for-like total, so there isn't one in the repo and there isn't one here. An earlier draft of this project did fuse two unrelated BEAM numbers into a headline that read well and did not exist. Once burned.
Genuinely depends, and I'll say it against my own interest:
Use the LLM-distilling architecture if your questions are heavy on temporal and multi-hop reasoning over dense material, a strong reader is doing the answering, per-memory cost is not a constraint, and sending content to a model provider is fine for you. It is better at that, measurably, and I've shown you the category where it beats me and why.
Use this one if the marginal cost of a memory has to be zero, your data can't leave, you need it to work offline, you want the retrieval path to be inspectable and switchable rather than a model's opinion, or if a system that confidently invents a satisfaction rate is a worse outcome for you than one that says nothing.
That's the trade. It isn't "we're better." It's that an LLM in the write path buys a better representation and charges you money, latency, and your data for it, and for a large number of real deployments that's the wrong purchase.
Parts 1–3 said the field measures accuracy on questions that have answers, that the standard harness for the most-quoted benchmark forbids abstaining, that when I built the missing metric my own system scored zero, and finally that the two public benchmarks disagree because each samples one point of a hidden axis, which turns a yes/no question into a coordinate.
This part says the rest of it: that the same design which loses temporal_reasoning
is the one that costs $0, ingests 4.3× faster, and never sends a document anywhere, and that the incumbent's better aggregate comes with a 46% fabrication rate on the questions where the honest answer is silence.
Both of those are true at once. Publishing only the flattering half is the thing this whole series is against.
On LOCOMO, same questions, same generator, same judge, only the memory differs. Paired McNemar over per-question outcomes, n=1,540:
| generator | judge | RE-call | Mem0 | paired p |
|---|---|---|---|---|
| gpt-4o-mini | gpt-4o-mini | 0.416 |
0.378 | 0.0059 | | gpt-4o-mini | gpt-4o | 0.466 | 0.412 | 0.00018 | | gpt-4o | gpt-4o | 0.484 | 0.444 | 0.0065 |
Holm–Bonferroni across all five cells run (largest adjusted p = 0.012), and it holds on the 1,369 questions where both judges agree (0.440 vs 0.399, p = 0.006).
Two things I'll say louder than the result. The lead is a property of the reader, not a universal fact. The margin shrinks as the generator strengthens and reverses on Claude Sonnet (0.565 vs 0.608, n=584, a generator run after pre-registration and labelled as such). And note the absolute numbers: 0.416–0.484, nothing like the 92.5 in the headlines, because a paired protocol with a strict judge and a matched retrieval budget measures something the leniency stack in Part 1 does not.
Everything above is a number I produced about my own system. You have no way to recompute it without buying inference.
That bothered me enough to build the fix: a benchmark where every number is recomputable by a hostile party for $0, with no model in the loop: no judge, no paid generator, git as the database, and disputing a result is a pull request. It has RE-call on its board with an unflattering score, and a deliberately fabricated submission committed as a fixture because it passes five of the six mechanical checks.
That's its own post, coming next, and it's the one I'd most like you to act on.
*Every figure: *
results/FINDINGS.md
§9d–§9p, results/RESULTS.md
§9–§11. RE-call is MIT, pip install recall-rag
.