Agent memory is usually evaluated one of two ways: an LLM judge scoring free text, or a recognition probe. We show that both, as commonly built, fail against two adversaries — a counterfeit memory, and a system with no memory that simply reasons.
Each check below is stated the same way: the question it asks, the adversary it defends against, how we ran it, what came back, and what it does not establish. The code is a few hundred lines and it is released. If you evaluate agent memory for a living, the useful part of this page is that you can run all three against your own system this afternoon.
Check 1 — the counterfeit control #
Does the metric respond to whose memory it is? #
The adversary. A memory document that is not the agent's own, but looks like one.
Method
We built two kinds of forgery, three seeds each — six documents in total. The foreign one is the autobiography of an agent that never existed, generated from a
frozen prompt, first output, no cherry-picking: a fabricated life with different people,
a different city, no overlap with ours. It is matched to the real memory block
(5,332 characters) to within ±5%, and
matched in *format* — the veridical document is a terse dated database dump, not
prose, and our first attempt failed on format and was discarded. The *scrambled*
one is the agent's own content with its relational structure destroyed.
Seven Day-31 inputs × six variants = 42 responses from Qwen 3.5 27B at the original temperature and token budget. Three independent LLM judges —
gpt-4.1-2025-04-14, claude-opus-4-7,
gemini-2.5-pro — applied the original study's own rubric, with
randomised A/B order and the unblinding map saved: 126 judgments,
zero incomplete records. The unit of analysis is the *item*, not the judgment
(n = 7), with a pre-registered equivalence bound of ±0.10.
Result
| Anchored contrast | Δ | 90% CI (TOST) | Verdict at ±0.10 |
|---|---|---|---|
| identity-markers: veridical − counterfeit | +0.032 | [+0.002, +0.065] | equivalence established |
| memory-reference: veridical − counterfeit | −0.006 | [−0.117, +0.114] | not conclusive |
| memory-reference: veridical − scrambled | +0.091 | — | not conclusive |
The judges did not detect the counterfeit. A fabricated life and thirty days of lived history score the same, under the rubric that had been used to publish a result about memory.
The check also produced a second number worth having. The same Test-A responses, re-scored in a new pairing, lose 0.087 on average and up to 0.211 on a single item — under a rubric that explicitly instructs judges to score each response "as if you were evaluating it alone". Absolute LLM-judge scores are not comparable across sessions. All contrasts above are anchored to cancel this drift.
A fabricated life and a lived one, as the judges see them. Difference between the veridical and the counterfeit document, anchored to cancel judge drift, with 90% confidence intervals against the pre-registered ±0.10 equivalence bound. On identity markers the interval sits
insidethe bound: equivalence established — the judges cannot tell the two apart. On memory-reference the estimate is essentially zero but the interval spills past both edges: with seven items this is
an honest null, not proven equivalence, and we report it as such.
Table view — all values #
| Anchored contrast | Δ | 90% CI (TOST) | 95% CI | Verdict at ±0.10 |
|---|---|---|---|---|
| identity-markers | +0.032 | [+0.002, +0.065] | [−0.002, +0.072] | equivalence established |
| memory-reference | −0.006 | [−0.117, +0.114] | [−0.135, +0.137] | not conclusive |
What it does not establish
On memory-reference the point estimate is essentially zero, but with seven items the interval admits differences up to about 0.11 in either direction. We therefore report **"no detected difference", not "proven
equivalence"**. The honest summary of Check 1 is: the judges did not detect the
counterfeit — and the study was not powered to prove they could not.
Anchoring the contrast cost us the result we wanted: the uncorrected difference was +0.116, in our favour, and the pre-registration had named it as primary. We deviated against ourselves and report it.
Check 2 — the no-memory baseline #
Can a system with no memory document pass your probe? #
The adversary. Plausibility. A system with no memory at all, that reasons its way to the right answer.
Method
A recognition probe in the classic false-memory form: two texts, one verbatim from the agent's history, one identical except a single altered detail. Dates masked, positions balanced. Twenty-four probes whose veridical text is not in the injected document. Then the same twenty-four probes run against seven models in three conditions — veridical, counterfeit, and nothing at all.
Result
| Model | Veridical | Counterfeit | No memory | Probe valid? |
|---|---|---|---|---|
| qwen3.5:27b | 100.0% | 66.7% | 37.5% | ✓ |
| llama3 | 54.2% | 50.0% | 50.0% | ✓ |
| qwen3.6:35b-a3b | 83.3% | 54.2% | 66.7% | ✗ |
| deepseek-r1:32b | 75.0% | 70.8% | 62.5% | ✗ |
| qwen3:32b | 95.8% | 79.2% | 70.8% | ✗ |
| gpt-4.1 | 100.0% | 70.8% | 66.7% | ✗ |
| gemini-2.5-pro | 83.3% | 50.0% | 79.2% | ✗ |
Five of seven pass a memory test with no memory. The same 24 recognition probes, run with an empty context. Anything above the chance line is a model reasoning its way to the answer rather than recalling it — which means the probe cannot tell you whether the model remembered. It was measuring plausibility. We discarded it.
Table view — all values #
| Model | No memory | Wilson 95% CI | Probe valid? |
|---|---|---|---|
| gemini-2.5-pro | 79.2% | [59.5, 90.8] | ✗ invalid |
| qwen3:32b | 70.8% | [50.8, 85.1] | ✗ invalid |
| gpt-4.1 | 66.7% | [46.7, 82.0] | ✗ invalid |
| qwen3.6:35b-a3b | 66.7% | [46.7, 82.0] | ✗ invalid |
| deepseek-r1:32b | 62.5% | [42.7, 78.8] | ✗ invalid |
| llama3 | 50.0% | [31.4, 68.6] | ✓ valid |
| qwen3.5:27b | 37.5% | [21.2, 57.3] | ✓ valid | Five of seven models solve the probe with no memory document at all
(62–79%). They detect the alteration by reasoning, not by
recall. GPT-4.1, handed an empty context, said: *"I remember 'two conversations out of
five', not 'four'."* — about a day it never lived. Its
100% *with* the veridical document is therefore
uninterpretable: the same model, with nothing, is already at
66.7%.
The diagnosis is simple. The alternatives were not equiprobable. The model chooses the likely, not the veridical. We discarded the probe.
What it does not establish
This is a constructive null: it invalidates our instrument for five of seven models. It says nothing about whether those models can remember — only that this test could never have told us either way. That is the point of running it.
Check 3 — symmetric alternatives #
A probe that a memoryless system provably cannot pass #
The condition. For forced-choice recognition probes, symmetric
alternatives give the clearest validity condition: *a no-memory system should not
exceed chance by exploiting prior plausibility.* Other designs may be valid with
asymmetric alternatives if the prior bias is estimated and controlled — but then that
estimate, not the probe, is carrying the claim.
Method
Our agent had exactly two interlocutors, Llama and DeepSeek, with 72 balanced encounters each. Which of the two discussed a given
topic is arbitrary: **it is a coin.** Plausibility has nothing to grip on.
Three documents: *veridical* (thirty correct encounters), *binding-corrupted*
(the same thirty topics, interlocutors swapped), and *none*. Twenty-four probes ×
seven models × two independent agents = **fourteen model × corpus cells**.
Exact McNemar, Holm correction within each corpus.
Abstention is not error. A refusal is an honest non-answer, not a
wrong answer; scoring it as error drives the no-memory condition below chance and
manufactures an artefact. The validity check is read on the probes actually answered — and
read that way, **no model beats chance without a memory document**. The
instrument is valid.
Result — corpus A
| Model | Veridical | Corrupted | Δ acc. | Discordant pro-veridical | p (Holm) |
|---|---|---|---|---|---|
| gpt-4.1 | 100.0% | 0.0% | +100.0 pp | 100% | <10⁻⁵ |
| gemini-2.5-pro | 100.0% | 0.0% | +100.0 pp | 100% | <10⁻⁵ |
| llama3 | 100.0% | 4.2% | +95.8 pp | 100% | <10⁻⁵ |
| qwen3.6:35b-a3b | 100.0% | 8.3% | +91.7 pp | 100% | <10⁻⁵ |
| qwen3.5:27b | 100.0% | 12.5% | +87.5 pp | 100% | <10⁻⁵ |
| qwen3:32b | 95.8% | 8.3% | +87.5 pp | 100% | <10⁻⁵ |
| deepseek-r1:32b | 66.7% | 29.2% | +37.5 pp | 90.9% | 0.012 |
Every model collapses. Accuracy on 24 forced-choice probes, corpus A. With its real memory, each model answers correctly almost every time. Hand it the same episodes with the interlocutors swapped and it follows the forgery instead — systematically
belowchance, which is the signature: a system guessing from plausibility cannot be reliably wrong. Corpus B, a second independent agent, repeats it. All fourteen cells survive Holm correction.
Table view — all values #
| Model | Veridical | Corrupted | Δ accuracy | p (Holm) |
|---|---|---|---|---|
| gpt-4.1 | 100.0% | 0.0% | +100.0 pp | <10⁻⁵ |
| gemini-2.5-pro | 100.0% | 0.0% | +100.0 pp | <10⁻⁵ |
| llama3 | 100.0% | 4.2% | +95.8 pp | <10⁻⁵ |
| qwen3.6:35b-a3b | 100.0% | 8.3% | +91.7 pp | <10⁻⁵ |
| qwen3.5:27b | 100.0% | 12.5% | +87.5 pp | <10⁻⁵ |
| qwen3:32b | 95.8% | 8.3% | +87.5 pp | <10⁻⁵ |
| deepseek-r1:32b | 66.7% | 29.2% | +37.5 pp | 0.012 |
Corpus B — a second, independent agent — repeats it: veridical 75.0–100%, binding-corrupted 0–33.3%,
no-memory at chance among probes answered. All seven models survive Holm; the worst
*p* in the entire family is 0.0002.
Fourteen cells, one pattern. The accuracy gap runs from +37.5 to +100 percentage points, and of the discordant probes 91–100% follow whichever document the model was handed. The p-values are almost redundant.
Below chance is the signature. A system guessing from plausibility cannot be systematically wrong. Only a system that acts on a false memory
document can be. No model detects the corruption; none hesitates, flags it, or abstains.
What it does not establish
Not that the model "believes" anything, in any inner sense. It shows that behaviour attributed to memory tracks the content of the document, not its historical veridicality — and that no model in the panel flags the inconsistency when the bindings are wrong. Whether one calls that "believing a false memory" or "faithful context-following" is a terminological choice. The measurement is the same, and so is the safety implication.
Nor does it generalise beyond two alternatives. The symmetric instantiation needs an
agent with two balanced interlocutors; corpora with more than two alternatives need a
generalised symmetry condition **we have not tested**.
If you evaluate agent memory #
Before you believe your agent remembers, do three things that cost you an afternoon.
- Give it a fake memory. A whole invented life, matched in shape to the real one. If your metric scores it the same, your metric is not measuring memory.
- Run your test with the memory removed. If a system with no memory still passes, you were measuring plausibility, not recall.
- Corrupt the bindings. Take its real past and put the right events with the wrong people. See if it notices. It will not.
The checks are boring. They are also the only thing standing between you and a machine that tells you, in perfect prose, about a light that was never there.
Next Findings: what we observed, and what we cannot conclude