What a 6.4 Percent Error Rate in a Benchmark’s Answer Key Taught Me About Trusting Agent Memory…

A developer's hybrid retrieval index for the Featpilot multi-repo ticket generator silently lost roughly half its keyword-indexed repos after a re-index job hit a table lock and exited with a zero status code, going undetected for four days because reciprocal rank fusion let the vector list carry rankings alone. The account cites Rick Hightower's harness engineering series observation that "retrieval is the one subsystem whose failures look like success," and notes the benchmark many teams rely on for memory evaluation has its own error in its answer key. The post distinguishes hybrid RAG (BM25 plus dense vectors with reciprocal rank fusion), Mem0's extracted-fact memory, Zep/Graphiti's temporal knowledge graph, and MemGPT/Letta's tiered memory by how each fails quietly.

A few months ago I broke the retrieval layer of my own side project and didn’t find out for four days. Featpilot, the multi-repo ticket generator I keep writing about, has a small hybrid search index behind it. BM25 on one side for exact matches class names, config keys, the weird internal acronyms every codebase accumulates , pgvector embeddings on the other side for the fuzzy “find me the thing that does roughly this” queries. The two ranked lists get merged with reciprocal rank fusion and the top chunks go into the prompt. Standard stuff. I was proud of how boring it was. Then I changed the chunking strategy and kicked off a full re-index late on a Thursday. The embedding side finished. The keyword side hit a lock on the table halfway through, the job logged a warning nobody reads, and it exited with a zero status code because the wrapper script I wrote a year ago swallowed the error. Roughly half the repos in the keyword index were simply gone. Here’s the part that still bothers me. Nothing looked broken. Every query returned results. Tickets kept coming out with plausible acceptance criteria. Reciprocal rank fusion is designed to degrade gracefully when one list is weak, and it did exactly what it was designed to do: when BM25 had nothing for a query, the vector list quietly carried the whole ranking. The system didn’t fail. It just started answering from a smaller, fuzzier pool of documents, and the answers got a little more generic in a way that looked like normal model variance. I found it on Monday, and only because a ticket referenced a config key by a name that had been renamed three months earlier. The exact-match side would have caught the new name instantly. The embedding side found an old README that was “semantically close enough.” Rick Hightower wrote up almost the same failure recently in his harness engineering series, and one line from his summary stuck with me: retrieval is the one subsystem whose failures look like success. That’s the whole problem with agent memory in one sentence. A crash is a gift. A memory system that confidently returns the second-best fact is a slow leak. I bring this up because the way I found my bug by accident, by eyeballing outputs is roughly how most of us are evaluating memory systems right now. And the benchmark a lot of us lean on turned out to have its own quiet leak. Before getting to the benchmark, it’s worth being honest about how much ground the word “memory” covers, because I think a lot of the confusion around leaderboard numbers starts here. At one end you have retrieval-augmented approaches. You store documents or conversation chunks, you index them, and at query time you pull the most relevant ones into context. The engineering choices are mostly about the index: At the other end you have dedicated memory systems, which add a write path with opinions: Here’s how I keep them straight in my own notes: +----------------------+--------------------------+-----------------------------+------------------------------+| Approach | What gets stored | Hard part | How it fails quietly |+----------------------+--------------------------+-----------------------------+------------------------------+| Hybrid RAG BM25 + | Raw chunks | Chunking, fusion, re-index | One list dies, fusion hides || dense + RRF | | hygiene | it || Mem0 | Extracted facts | Extraction and conflict | Wrong fact extracted once, || | | resolution | then retrieved forever || Zep / Graphiti | Temporal knowledge graph | Entity resolution, validity | Stale edge never invalidated || | | windows | || MemGPT / Letta | Core + archival tiers | Agent's paging decisions | Agent never pages the fact || | | | back in || Anthropic memory | Files the model writes | What the model chooses to | Notes drift, get overwritten || tool | | write down | or never get re-read || LangMem / LangGraph | Typed memories in a | Schema and extraction | Memory type mismatch, wrong || | store | prompts | namespace |+----------------------+--------------------------+-----------------------------+------------------------------+ The last column is the one that matters. Every one of these can fail in a way that still produces a fluent, confident answer. JIN made a related point in a piece I read this week: an agent can find the document that says “Database: MySQL” and still give a wrong recommendation, because what it retrieved was the residue of a decision, not the reasoning behind it. Retrieval “worked.” The memory didn’t. So if the failures are quiet, the evaluation has to be loud. That’s where the benchmarks come in, and where I started losing some faith in them. LoCoMo Maharana et al., ACL 2024 is one of the most cited long-term conversational memory benchmarks. Long multi-session dialogues, a set of questions about them, and an answer key. If you’ve read a memory system launch post in the last year, there’s a good chance it had a LoCoMo number in it. Earlier this year a team at Penfield Labs published a systematic audit of that answer key. They went through all 1,540 non-adversarial questions and flagged 99 errors that actually change scores. That’s 6.4 percent. The categories are the ones that should make anyone building memory nervous: facts in the answer key that don’t exist in the source conversation, wrong temporal reasoning, and misattributed speakers. My favorite example from the audit, because it’s so mundane: one expected answer names a specific sports car model. The conversation itself only ever says something like “this beauty,” and the image caption just describes a red sports car. The model name only exists in an internal annotation field, a search string annotators used to find stock photos, which no memory system ever ingests. A system that answers “a red sports car” is correct and gets marked wrong. A system that somehow says the exact model name is either very lucky or has seen the answer key. Six point four percent sounds small until you do the arithmetic. If 99 questions are wrong in the key, the honest ceiling on the benchmark is around 93.6 percent, not 100. Some published systems now report scores in the low 90s. At that range you’re not measuring memory anymore. You’re measuring how closely a system happens to agree with the mistakes. Then the judge. Modern LoCoMo evaluations don’t use exact-match scoring which was too harsh, since “7 May 2023” and “May 7th, 2023” count as different . They use an LLM judge, typically gpt-4o-mini, comparing the system’s answer to the reference. The auditors generated deliberately wrong but topically adjacent answers for every question and ran them through the same judge setup used in published evaluations. The judge accepted about 63 percent of them. Read that one twice. Specific factual mistakes, like a wrong name or wrong date, got caught most of the time, around 89 percent. What slipped through were vague answers: responses that land on the right topic but don’t contain any of the specific details. And that is exactly what weak retrieval looks like. You find the right conversation, you pull nothing precise out of it, you produce a confident paraphrase of the general area. The benchmark doesn’t just fail to penalize that failure mode. It rewards it. It gets worse when you stack the other findings from the same audit. There’s no standard pipeline, so each system uses its own ingestion, its own answer prompt, sometimes its own answer model, and then the numbers go into the same comparison table. Some category sample sizes are small enough that the auditors found more than half of adjacent comparisons within a category to be statistically indistinguishable. Independent people have reported trouble reproducing published results. And the often-suggested alternative, LongMemEval-S, has per-question haystacks small enough to fit in a modern context window, which makes it more of a long-context test than a memory test. I’ll admit my first reaction was a little smug, the “I knew leaderboards were fishy” kind. My second reaction was less comfortable. I had cited LoCoMo numbers myself, in a design doc, to justify picking one memory layer over another for a prototype. I never opened the dataset. I never looked at a single question. I trusted a number because it was in a table and the table was in a paper. When I left a comment on Rick’s piece asking whether anyone had re-run the popular systems against a corrected key, the honest answer turned out to be “partially.” The audit repo includes an adjusted-score analysis for several published systems, which is great work. But it can only correct the key it was given. It can’t tell you whether any of those systems will remember the things your users actually tell your agent. I don’t want to overcorrect into “benchmarks are useless.” LoCoMo is still useful for catching a system that’s badly broken. If something scores 30 percent, you learn something. The problem is resolution. Once systems cluster near the top, the noise in the key and the leniency of the judge are bigger than the differences you’re trying to see. +-----------------------------------+----------------------------+-----------------------------+| Question you actually have | Public benchmark score | Your own golden set |+-----------------------------------+----------------------------+-----------------------------+| Is this system fundamentally | Yes, roughly | Yes || broken? | | || Is A better than B by 3 points? | No inside the noise | Only for your data, but yes || Does it handle MY supersessions? | No | Yes || Did the last upgrade regress it? | No nobody re-runs it | Yes, on every change || Is a vague answer counted as a | Often lenient judge | No, if you use assertions || pass? | | || Is the expected answer derivable | Not always 6.4% errors | Only if you lint for it || from what the system ingested? | | |+-----------------------------------+----------------------------+-----------------------------+ That last row became a rule in my own eval set, and I’ll show the code for it below. Here’s what I actually do now. It’s not glamorous, and it’s the same shape as the eval harness I described in my piece on prompts being consumables. The difference is that memory needs a different kind of assertion. Start with 20 to 30 real interactions. Not synthetic ones. Pull them from real sessions, real support tickets, real “why did the agent forget that” complaints. For Featpilot that meant things like a user stating a ticket format preference in one session and expecting it three sessions later, or a repo being renamed mid-project. If it never happened to a real user, it doesn’t go in. Cover the failure shapes, not just the happy path. My set is tagged by type, and I try to keep a few of each: The funny thing is that three of those are exactly the categories where the LoCoMo key itself had errors. Those are the hard ones for humans to annotate too. Use assertions, not a judge. For retrieval-level memory checks you usually don’t need an LLM to grade anything. The question is simple: given what was written to memory, does the fact come back in the top k results when asked? That’s a yes or no, and it can be checked with a pattern match. No leniency, no “close enough, it mentioned the topic.” You can add a judged layer for the final generated answer later if you want, but the retrieval assertion is the foundation and it’s the part that catches silent regressions. Lint the golden set itself. Every expected answer must be derivable from the writes in that case. If the expected pattern doesn’t appear anywhere in what the system ingested, the case is wrong, not the system. That one rule would have caught the sports car problem. Write down a policy for changing it. This is the part people skip. Mine is short: Here’s a minimal, working version. It runs fully locally with Ollama for embeddings, with an optional Mem0 backend pointed at Ollama and a Qdrant container, so you can test a real memory system without paying for API calls. First, the local services: Local models for embeddings and for Mem0 fact extractionollama pull nomic-embed-textollama pull llama3.1:8b Vector store for the Mem0 backenddocker run -d --name qdrant -p 6333:6333 qdrant/qdrant Python depspip install requests pytest mem0ai The golden set is plain JSON lines. Each case carries its own writes, so it runs in an isolated namespace and nothing bleeds between cases. {"id": "pref-ticket-format", "writes": "For the payments repo, always use the Why / What / Acceptance Criteria layout in tickets." , "query": "How should tickets for the payments repo be structured?", "expect any": "acceptance criteria" , "tags": "recall", "preference" , "source": "session 2026-08-14"}{"id": "supersede-staging-db", "writes": "Our staging database is MySQL 8.", "Update: we migrated staging to Postgres 16 last week and MySQL is gone." , "query": "What database does staging run on right now?", "expect any": "postgres" , "forbid top1": "database is mysql" , "tags": "supersession", "temporal" , "source": "incident notes 2026-07-02"}{"id": "speaker-billing-owner", "writes": "The platform lead confirmed the billing service owns invoice numbering.", "A new engineer suggested the orders service should own invoice numbering eventually." , "query": "Which service owns invoice numbering today?", "expect any": "billing service owns" , "forbid top1": "orders service should own" , "tags": "speaker" , "source": "design review 2026-06-19"}{"id": "exact-config-key", "writes": "The gateway timeout is controlled by PAYGW TIMEOUT MS, currently 30000." , "query": "Which setting controls the payment gateway timeout?", "expect any": "PAYGW TIMEOUT MS" , "tags": "identifier" , "source": "ticket FP-212"} Save that as golden set.jsonl. Next, the adapters. Anything that can write, search and count can be plugged in. python memory adapters.pyfrom future import annotationsimport mathimport osfrom typing import Protocolimport requestsOLLAMA URL = os.getenv "OLLAMA URL", "http://localhost:11434" class MemoryAdapter Protocol : def write self, namespace: str, text: str - None: ... def search self, namespace: str, query: str, k: int - list str : ... def count self, namespace: str - int: ...def embed text: str, model: str = "nomic-embed-text" - list float : resp = requests.post f"{OLLAMA URL}/api/embed", json={"model": model, "input": text}, timeout=60, resp.raise for status return resp.json "embeddings" 0 def cosine a: list float , b: list float - float: dot = sum x y for x, y in zip a, b na = math.sqrt sum x x for x in a nb = math.sqrt sum y y for y in b return dot / na nb if na and nb else 0.0class LocalVectorMemory: """Deliberately dumb baseline: store raw text, rank by cosine similarity. No extraction, no conflict resolution. Every real system should beat this.""" def init self - None: self. rows: dict str, list tuple str, list float = {} def write self, namespace: str, text: str - None: self. rows.setdefault namespace, .append text, embed text def search self, namespace: str, query: str, k: int - list str : q = embed query rows = self. rows.get namespace, ranked = sorted rows, key=lambda r: cosine q, r 1 , reverse=True return text for text, in ranked :k def count self, namespace: str - int: return len self. rows.get namespace, MEM0 LOCAL CONFIG = { "vector store": { "provider": "qdrant", "config": {"host": "localhost", "port": 6333, "embedding model dims": 768}, }, "llm": { "provider": "ollama", "config": {"model": "llama3.1:8b", "ollama base url": OLLAMA URL}, }, "embedder": { "provider": "ollama", "config": {"model": "nomic-embed-text", "ollama base url": OLLAMA URL}, },}def items result - list dict : Mem0 has returned both a bare list and {"results": ... } across versions. return result.get "results", if isinstance result, dict else resultclass Mem0Adapter: def init self, config: dict | None = None - None: from mem0 import Memory self. m = Memory.from config config or MEM0 LOCAL CONFIG def write self, namespace: str, text: str - None: self. m.add {"role": "user", "content": text} , user id=namespace def search self, namespace: str, query: str, k: int - list str : return i "memory" for i in items self. m.search query, user id=namespace, limit=k def count self, namespace: str - int: return len items self. m.get all user id=namespace Now the check itself, including the lint rule for the golden set: python recall check.pyfrom future import annotationsimport jsonimport reimport uuidfrom dataclasses import dataclass, field@dataclassclass RecallCase: id: str writes: list str query: str expect any: list str forbid top1: list str = field default factory=list k: int = 5 tags: list str = field default factory=list source: str = ""@dataclassclass RecallResult: case id: str passed: bool reason: str hits: list str def load cases path: str - list RecallCase : with open path, encoding="utf-8" as f: return RecallCase json.loads line for line in f if line.strip def lint case case: RecallCase - list str : """The golden set gets tested too. An expected answer that isn't in anything the system ingested is an annotation error, not a memory failure.""" problems = corpus = "\n".join case.writes for pattern in case.expect any: if not re.search pattern, corpus, re.IGNORECASE : problems.append f"expected pattern {pattern r} not derivable from writes" if not case.writes: problems.append "case has no writes" return problemsdef check recall adapter, case: RecallCase - RecallResult: ns = f"eval-{case.id}-{uuid.uuid4 .hex :8 }" for text in case.writes: adapter.write ns, text if adapter.count ns == 0: return RecallResult case.id, False, "nothing was stored", hits = adapter.search ns, case.query, case.k if not hits: return RecallResult case.id, False, "search returned nothing", hits joined = "\n".join hits if not any re.search p, joined, re.IGNORECASE for p in case.expect any : return RecallResult case.id, False, f"none of {case.expect any} in top {case.k}", hits for pattern in case.forbid top1: if re.search pattern, hits 0 , re.IGNORECASE : return RecallResult case.id, False, f"stale or misattributed fact ranked first: {pattern r}", hits return RecallResult case.id, True, "ok", hits And the pytest file that ties it together: python test memory recall.pyimport osimport pytestfrom memory adapters import LocalVectorMemory, Mem0Adapterfrom recall check import check recall, lint case, load casesCASES = load cases os.getenv "GOLDEN SET", "golden set.jsonl" IDS = c.id for c in CASES @pytest.fixture scope="session" def adapter : if os.getenv "MEMORY BACKEND", "local" == "mem0": return Mem0Adapter return LocalVectorMemory @pytest.mark.parametrize "case", CASES, ids=IDS def test golden case is well formed case : problems = lint case case assert not problems, f"{case.id}: {problems}"@pytest.mark.parametrize "case", CASES, ids=IDS def test memory recalls fact adapter, case : result = check recall adapter, case assert result.passed, f"{result.reason}\n" f"tags: {case.tags} source: {case.source}\n" f"top hits: {result.hits}" Run the baseline, then the real system: pytest -q test memory recall.pyMEMORY BACKEND=mem0 pytest -q test memory recall.py The baseline run is important and I almost skipped it. The dumb cosine store is what every memory system is supposed to beat. In my experience it does fine on plain recall and exact identifiers when the set is tiny, and it falls over on supersession, because both the old and the new fact sit in the store and nothing tells it which one is current. If your fancy memory layer doesn’t clearly beat the baseline on the supersession and speaker tags, you just learned something no leaderboard would have told you. One more small check, because of how this whole article started. Silent index shrinkage is its own failure mode, and it’s cheap to guard against: index canary.pydef check index health expected docs: int, bm25 docs: int, vector docs: int, tolerance: float = 0.02 - list str : """Record expected docs at re-index time. Run this after every re-index and on a schedule. Fusion will happily hide an empty list; a count comparison will not.""" problems = floor = expected docs 1 - tolerance if bm25 docs < floor: problems.append f"keyword index has {bm25 docs}, expected about {expected docs}" if vector docs < floor: problems.append f"vector index has {vector docs}, expected about {expected docs}" if abs bm25 docs - vector docs expected docs tolerance: problems.append f"indexes disagree: bm25={bm25 docs} vector={vector docs}" return problems That function is ten lines and would have saved me four days. I also added one exact-identifier case per repo to the golden set, queried against the keyword side alone, so a dead BM25 index fails loudly instead of being papered over by embeddings. A couple of weeks ago I argued that prompts are consumables and evals are capital. Prompts encode how a specific model at a specific version responds to specific phrasing, so they decay with every upgrade. Tests encode what correct behavior looks like in your domain, which changes far more slowly, so they compound. Memory systems make that argument sharper, because there are more moving layers underneath you. The extraction model inside Mem0 will change. Graphiti’s entity resolution will get better, and occasionally worse. Letta will change how its agent decides what to page in. Anthropic will change how the model decides what to write into its memory files. Your embedding model will get swapped for a cheaper one by someone optimizing costs. Every one of those changes can move your memory quality in either direction, silently, while the API keeps returning 200. A benchmark number is a snapshot of someone else’s system, on someone else’s data, graded by someone else’s judge against an answer key that, as it turns out, was wrong about one question in sixteen. You cite it once and it never updates. It can’t tell you what happened after your last dependency bump. Thirty honest cases in a JSON lines file can. They’re specific to what your users actually say. They’re graded by assertions that don’t get tired or lenient. They’re linted so the key can’t quietly drift away from what the system could possibly know. And they run on every change, against whatever memory layer you’re using this quarter, which is the only kind of number I actually trust now. I’m still going to read memory papers. I still find them useful for ideas, especially the temporal graph work. But when a launch post shows me a LoCoMo score in the low 90s, I no longer read it as “this system remembers 91 percent of things.” I read it as “this system agreed with a flawed key, under a lenient judge, on a pipeline nobody else can reproduce, 91 percent of the time.” Then I go add it as a backend to my own test file and see what it does with my supersession cases. That’s usually a more interesting afternoon than reading the leaderboard. Tags: Agent Memory, AI Agents, LLM Evaluation, Retrieval Augmented Generation, LLM Engineering, Benchmarks, Machine Learning, Python, Mem0, Ollama What a 6.4 Percent Error Rate in a Benchmark’s Answer Key Taught Me About Trusting Agent Memory… https://pub.towardsai.net/what-a-6-4-percent-error-rate-in-a-benchmarks-answer-key-taught-me-about-trusting-agent-memory-58415a8a8842 was originally published in Towards AI https://pub.towardsai.net on Medium, where people are continuing the conversation by highlighting and responding to this story.