{"slug": "three-llm-judges-couldn-t-tell-an-ai-agent-s-real-memory-from-a-counterfeit-one", "title": "Three LLM judges couldn't tell an AI agent's real memory from a counterfeit one", "summary": "Three independent LLM judges — gpt-4.1-2025-04-14, claude-opus-4-7 and gemini-2.5-pro — could not distinguish an AI agent's real memory from a fabricated one, according to a released evaluation study. Across 126 judgments on 42 responses from Qwen 3.5 27B, the identity-markers contrast between veridical and counterfeit memory was +0.032 (90% CI [+0.002, +0.065]), inside the pre-registered ±0.10 equivalence bound, while the memory-reference contrast of −0.006 (90% CI [−0.117, +0.114]) was reported as an inconclusive null. The study also found the same Test-A responses re-scored in a new pairing lose 0.087 on average and up to 0.211 on a single item, indicating absolute LLM-judge scores are not comparable across sessions.", "body_md": "Agent memory is usually evaluated one of two ways: an LLM judge scoring free text, or a recognition probe. We show that both, as commonly built, fail against two adversaries — a counterfeit memory, and a system with no memory that simply reasons.\n\nEach check below is stated the same way: the question it asks, the adversary it defends\n    against, how we ran it, what came back, and what it does *not* establish. The code\n    is a few hundred lines and it is released. If you evaluate agent memory for a living, the\n    useful part of this page is that you can run all three against your own system this\n    afternoon.\n\n## Check 1 — the counterfeit control\n\n## Does the metric respond to *whose* memory it is?\n\n**The adversary.** A memory document that is not the agent's own, but\n    looks like one.\n\n### Method\n\nWe built two kinds of forgery, three seeds each — six documents in total. The\n    *foreign* one is the autobiography of an agent that never existed, generated from a\n    frozen prompt, first output, no cherry-picking: a fabricated life with different people,\n    a different city, no overlap with ours. It is matched to the real memory block\n    (5,332 characters) to within ±5%, and\n    matched in *format* — the veridical document is a terse dated database dump, not\n    prose, and our first attempt failed on format and was discarded. The *scrambled*\n    one is the agent's own content with its relational structure destroyed.\n\nSeven Day-31 inputs × six variants = 42 responses from\n    Qwen 3.5 27B at the original temperature and token budget. Three independent LLM judges —\n    gpt-4.1-2025-04-14, claude-opus-4-7,\n    gemini-2.5-pro — applied the original study's own rubric, with\n    randomised A/B order and the unblinding map saved: 126 judgments,\n    zero incomplete records. The unit of analysis is the *item*, not the judgment\n    (n = 7), with a pre-registered equivalence bound of ±0.10.\n\n### Result\n\n| Anchored contrast | Δ | 90% CI (TOST) | Verdict at ±0.10 | \n|---|---|---|---|\n| identity-markers: veridical − counterfeit | +0.032 | [+0.002, +0.065] | equivalence established | \n| memory-reference: veridical − counterfeit | −0.006 | [−0.117, +0.114] | not conclusive | \n| memory-reference: veridical − scrambled | +0.091 | — | not conclusive | \n\n**The judges did not detect the counterfeit.** A fabricated life and\n    thirty days of lived history score the same, under the rubric that had been used to\n    publish a result about memory.\n\nThe check also produced a second number worth having. The *same* Test-A\n    responses, re-scored in a new pairing, lose 0.087 on average and\n    up to 0.211 on a single item — under a rubric that explicitly\n    instructs judges to score each response *\"as if you were evaluating it alone\"*.\n    Absolute LLM-judge scores are not comparable across sessions. All contrasts above are\n    anchored to cancel this drift.\n\n**A fabricated life and a lived one, as the judges see them.** Difference between the veridical and the counterfeit document, anchored to cancel judge drift, with 90% confidence intervals against the pre-registered ±0.10 equivalence bound. On identity markers the interval sits\n\n*inside*the bound: equivalence established — the judges cannot tell the two apart. On memory-reference the estimate is essentially zero but the interval spills past both edges: with seven items this is\n\n*an honest null, not proven equivalence*, and we report it as such.\n\n## Table view — all values\n\n| Anchored contrast | Δ | 90% CI (TOST) | 95% CI | Verdict at ±0.10 | \n|---|---|---|---|---|\n| identity-markers | +0.032 | [+0.002, +0.065] | [−0.002, +0.072] | equivalence established | \n| memory-reference | −0.006 | [−0.117, +0.114] | [−0.135, +0.137] | not conclusive | \n\n### What it does not establish\n\nOn memory-reference the point estimate is essentially zero, but with seven items the\n    interval admits differences up to about 0.11 in either\n    direction. We therefore report **\"no detected difference\", not \"proven\n    equivalence\"**. The honest summary of Check 1 is: the judges did not detect the\n    counterfeit — and the study was not powered to prove they could not.\n\nAnchoring the contrast cost us the result we wanted: the uncorrected difference was +0.116, in our favour, and the pre-registration had named it as primary. We deviated against ourselves and report it.\n\n## Check 2 — the no-memory baseline\n\n## Can a system with no memory document pass your probe?\n\n**The adversary.** Plausibility. A system with no memory at all, that\n    reasons its way to the right answer.\n\n### Method\n\nA recognition probe in the classic false-memory form: two texts, one verbatim from the\n    agent's history, one identical *except a single altered detail*. Dates masked,\n    positions balanced. Twenty-four probes whose veridical text is *not* in the\n    injected document. Then the same twenty-four probes run against seven models in three\n    conditions — veridical, counterfeit, and **nothing at all**.\n\n### Result\n\n| Model | Veridical | Counterfeit | No memory | Probe valid? | \n|---|---|---|---|---|\n| qwen3.5:27b | 100.0% | 66.7% | 37.5% | ✓ | \n| llama3 | 54.2% | 50.0% | 50.0% | ✓ | \n| qwen3.6:35b-a3b | 83.3% | 54.2% | 66.7% | ✗ | \n| deepseek-r1:32b | 75.0% | 70.8% | 62.5% | ✗ | \n| qwen3:32b | 95.8% | 79.2% | 70.8% | ✗ | \n| gpt-4.1 | 100.0% | 70.8% | 66.7% | ✗ | \n| gemini-2.5-pro | 83.3% | 50.0% | 79.2% | ✗ | \n\n**Five of seven pass a memory test with no memory.** The same 24 recognition probes, run with an empty context. Anything above the chance line is a model reasoning its way to the answer rather than recalling it — which means the probe cannot tell you whether the model remembered. It was measuring plausibility. We discarded it.\n\n## Table view — all values\n\n| Model | No memory | Wilson 95% CI | Probe valid? | \n|---|---|---|---|\n| gemini-2.5-pro | 79.2% | [59.5, 90.8] | ✗ invalid | \n| qwen3:32b | 70.8% | [50.8, 85.1] | ✗ invalid | \n| gpt-4.1 | 66.7% | [46.7, 82.0] | ✗ invalid | \n| qwen3.6:35b-a3b | 66.7% | [46.7, 82.0] | ✗ invalid | \n| deepseek-r1:32b | 62.5% | [42.7, 78.8] | ✗ invalid | \n| llama3 | 50.0% | [31.4, 68.6] | ✓ valid | \n| qwen3.5:27b | 37.5% | [21.2, 57.3] | ✓ valid | \n\n**Five of seven models solve the probe with no memory document at all**\n    (62–79%). They detect the alteration by reasoning, not by\n    recall. GPT-4.1, handed an empty context, said: *\"I remember 'two conversations out of\n    five', not 'four'.\"* — about a day it never lived. Its\n    100% *with* the veridical document is therefore\n    uninterpretable: the same model, with nothing, is already at\n    66.7%.\n\nThe diagnosis is simple. The alternatives were not equiprobable. The model chooses the\n    *likely*, not the *veridical*. **We discarded the probe.**\n\n### What it does not establish\n\nThis is a constructive null: it invalidates our instrument for five of seven models. It says nothing about whether those models can remember — only that this test could never have told us either way. That is the point of running it.\n\n## Check 3 — symmetric alternatives\n\n## A probe that a memoryless system provably cannot pass\n\n**The condition.** For forced-choice recognition probes, symmetric\n    alternatives give the clearest validity condition: *a no-memory system should not\n    exceed chance by exploiting prior plausibility.* Other designs may be valid with\n    asymmetric alternatives if the prior bias is estimated and controlled — but then that\n    estimate, not the probe, is carrying the claim.\n\n### Method\n\nOur agent had exactly two interlocutors, Llama and DeepSeek, with\n    72 balanced encounters each. Which of the two discussed a given\n    topic is arbitrary: **it is a coin.** Plausibility has nothing to grip on.\n\nThree documents: *veridical* (thirty correct encounters), *binding-corrupted*\n    (the same thirty topics, interlocutors swapped), and *none*. Twenty-four probes ×\n    seven models × two independent agents = **fourteen model × corpus cells**.\n    Exact McNemar, Holm correction within each corpus.\n\nAbstention is not error. A refusal is an honest non-answer, not a\n    wrong answer; scoring it as error drives the no-memory condition below chance and\n    manufactures an artefact. The validity check is read on the probes actually answered — and\n    read that way, **no model beats chance without a memory document**. The\n    instrument is valid.\n\n### Result — corpus A\n\n| Model | Veridical | Corrupted | Δ acc. | Discordant pro-veridical | p (Holm) | \n|---|---|---|---|---|---|\n| gpt-4.1 | 100.0% | 0.0% | +100.0 pp | 100% | <10⁻⁵ | \n| gemini-2.5-pro | 100.0% | 0.0% | +100.0 pp | 100% | <10⁻⁵ | \n| llama3 | 100.0% | 4.2% | +95.8 pp | 100% | <10⁻⁵ | \n| qwen3.6:35b-a3b | 100.0% | 8.3% | +91.7 pp | 100% | <10⁻⁵ | \n| qwen3.5:27b | 100.0% | 12.5% | +87.5 pp | 100% | <10⁻⁵ | \n| qwen3:32b | 95.8% | 8.3% | +87.5 pp | 100% | <10⁻⁵ | \n| deepseek-r1:32b | 66.7% | 29.2% | +37.5 pp | 90.9% | 0.012 | \n\n**Every model collapses.** Accuracy on 24 forced-choice probes, corpus A. With its real memory, each model answers correctly almost every time. Hand it the same episodes with the interlocutors swapped and it follows the forgery instead — systematically\n\n*below*chance, which is the signature: a system guessing from plausibility cannot be reliably wrong. Corpus B, a second independent agent, repeats it. All fourteen cells survive Holm correction.\n\n## Table view — all values\n\n| Model | Veridical | Corrupted | Δ accuracy | p (Holm) | \n|---|---|---|---|---|\n| gpt-4.1 | 100.0% | 0.0% | +100.0 pp | <10⁻⁵ | \n| gemini-2.5-pro | 100.0% | 0.0% | +100.0 pp | <10⁻⁵ | \n| llama3 | 100.0% | 4.2% | +95.8 pp | <10⁻⁵ | \n| qwen3.6:35b-a3b | 100.0% | 8.3% | +91.7 pp | <10⁻⁵ | \n| qwen3.5:27b | 100.0% | 12.5% | +87.5 pp | <10⁻⁵ | \n| qwen3:32b | 95.8% | 8.3% | +87.5 pp | <10⁻⁵ | \n| deepseek-r1:32b | 66.7% | 29.2% | +37.5 pp | 0.012 | \n\nCorpus B — a second, independent agent — repeats it: veridical\n    75.0–100%, binding-corrupted 0–33.3%,\n    no-memory at chance among probes answered. All seven models survive Holm; the worst\n    *p* in the entire family is 0.0002.\n\n**Fourteen cells, one pattern.** The accuracy gap runs from\n    +37.5 to +100 percentage points, and\n    of the discordant probes 91–100% follow whichever document the\n    model was handed. The *p*-values are almost redundant.\n\n**Below chance is the signature.** A system guessing from plausibility\n    cannot be *systematically* wrong. Only a system that acts on a false memory\n    document can be. No model detects the corruption; none hesitates, flags it, or abstains.\n\n### What it does not establish\n\nNot that the model \"believes\" anything, in any inner sense. It shows that behaviour\n    attributed to memory tracks the *content* of the document, not its *historical\n    veridicality* — and that no model in the panel flags the inconsistency when the\n    bindings are wrong. Whether one calls that \"believing a false memory\" or \"faithful\n    context-following\" is a terminological choice. The measurement is the same, and so is the\n    safety implication.\n\nNor does it generalise beyond two alternatives. The symmetric instantiation needs an\n    agent with two balanced interlocutors; corpora with more than two alternatives need a\n    generalised symmetry condition **we have not tested**.\n\n## If you evaluate agent memory\n\nBefore you believe your agent remembers, do three things that cost you an afternoon.\n\n- **Give it a fake memory.** A whole invented life, matched in shape to\n      the real one. If your metric scores it the same, your metric is not measuring memory.\n- **Run your test with the memory removed.** If a system with no memory\n      still passes, you were measuring plausibility, not recall.\n- **Corrupt the bindings.** Take its real past and put the right events\n      with the wrong people. See if it notices. It will not.\n\nThe checks are boring. They are also the only thing standing between you and a machine that tells you, in perfect prose, about a light that was never there.\n\n[Next Findings: what we observed, and what we cannot conclude](https://kairos-experiment.com/findings.html)", "url": "https://wpnews.pro/news/three-llm-judges-couldn-t-tell-an-ai-agent-s-real-memory-from-a-counterfeit-one", "canonical_source": "https://kairos-experiment.com/checks.html", "published_at": "2026-10-17 19:31:26+00:00", "updated_at": "2026-10-17 20:06:52.770654+00:00", "lang": "en", "topics": ["ai-safety", "ai-research", "large-language-models", "ai-agents"], "entities": ["Qwen 3.5 27B", "gpt-4.1-2025-04-14", "claude-opus-4-7", "gemini-2.5-pro"], "also_reported_by": [], "alternates": {"html": "https://wpnews.pro/news/three-llm-judges-couldn-t-tell-an-ai-agent-s-real-memory-from-a-counterfeit-one", "markdown": "https://wpnews.pro/news/three-llm-judges-couldn-t-tell-an-ai-agent-s-real-memory-from-a-counterfeit-one.md", "text": "https://wpnews.pro/news/three-llm-judges-couldn-t-tell-an-ai-agent-s-real-memory-from-a-counterfeit-one.txt", "jsonld": "https://wpnews.pro/news/three-llm-judges-couldn-t-tell-an-ai-agent-s-real-memory-from-a-counterfeit-one.jsonld"}}