{"slug": "what-a-6-4-percent-error-rate-in-a-benchmarks-answer-key-taught-me-about-agent", "title": "What a 6.4 Percent Error Rate in a Benchmark’s Answer Key Taught Me About Trusting Agent Memory…", "summary": "A developer's hybrid retrieval index for the Featpilot multi-repo ticket generator silently lost roughly half its keyword-indexed repos after a re-index job hit a table lock and exited with a zero status code, going undetected for four days because reciprocal rank fusion let the vector list carry rankings alone. The account cites Rick Hightower's harness engineering series observation that \"retrieval is the one subsystem whose failures look like success,\" and notes the benchmark many teams rely on for memory evaluation has its own error in its answer key. The post distinguishes hybrid RAG (BM25 plus dense vectors with reciprocal rank fusion), Mem0's extracted-fact memory, Zep/Graphiti's temporal knowledge graph, and MemGPT/Letta's tiered memory by how each fails quietly.", "body_md": "A few months ago I broke the retrieval layer of my own side project and didn’t find out for four days.\n\nFeatpilot, the multi-repo ticket generator I keep writing about, has a small hybrid search index behind it. BM25 on one side for exact matches (class names, config keys, the weird internal acronyms every codebase accumulates), pgvector embeddings on the other side for the fuzzy “find me the thing that does roughly this” queries. The two ranked lists get merged with reciprocal rank fusion and the top chunks go into the prompt. Standard stuff. I was proud of how boring it was.\n\nThen I changed the chunking strategy and kicked off a full re-index late on a Thursday. The embedding side finished. The keyword side hit a lock on the table halfway through, the job logged a warning nobody reads, and it exited with a zero status code because the wrapper script I wrote a year ago swallowed the error. Roughly half the repos in the keyword index were simply gone.\n\nHere’s the part that still bothers me. Nothing looked broken. Every query returned results. Tickets kept coming out with plausible acceptance criteria. Reciprocal rank fusion is designed to degrade gracefully when one list is weak, and it did exactly what it was designed to do: when BM25 had nothing for a query, the vector list quietly carried the whole ranking. The system didn’t fail. It just started answering from a smaller, fuzzier pool of documents, and the answers got a little more generic in a way that looked like normal model variance.\n\nI found it on Monday, and only because a ticket referenced a config key by a name that had been renamed three months earlier. The exact-match side would have caught the new name instantly. The embedding side found an old README that was “semantically close enough.”\n\nRick Hightower wrote up almost the same failure recently in his harness engineering series, and one line from his summary stuck with me: retrieval is the one subsystem whose failures look like success. That’s the whole problem with agent memory in one sentence. A crash is a gift. A memory system that confidently returns the second-best fact is a slow leak.\n\nI bring this up because the way I found my bug (by accident, by eyeballing outputs) is roughly how most of us are evaluating memory systems right now. And the benchmark a lot of us lean on turned out to have its own quiet leak.\n\nBefore getting to the benchmark, it’s worth being honest about how much ground the word “memory” covers, because I think a lot of the confusion around leaderboard numbers starts here.\n\nAt one end you have retrieval-augmented approaches. You store documents or conversation chunks, you index them, and at query time you pull the most relevant ones into context. The engineering choices are mostly about the index:\n\nAt the other end you have dedicated memory systems, which add a write path with opinions:\n\nHere’s how I keep them straight in my own notes:\n\n```\n+----------------------+--------------------------+-----------------------------+------------------------------+| Approach             | What gets stored         | Hard part                   | How it fails quietly         |+----------------------+--------------------------+-----------------------------+------------------------------+| Hybrid RAG (BM25 +   | Raw chunks               | Chunking, fusion, re-index  | One list dies, fusion hides  || dense + RRF)         |                          | hygiene                     | it                           || Mem0                 | Extracted facts          | Extraction and conflict     | Wrong fact extracted once,   ||                      |                          | resolution                  | then retrieved forever       || Zep / Graphiti       | Temporal knowledge graph | Entity resolution, validity | Stale edge never invalidated ||                      |                          | windows                     |                              || MemGPT / Letta       | Core + archival tiers    | Agent's paging decisions    | Agent never pages the fact   ||                      |                          |                             | back in                      || Anthropic memory     | Files the model writes   | What the model chooses to   | Notes drift, get overwritten || tool                 |                          | write down                  | or never get re-read         || LangMem / LangGraph  | Typed memories in a      | Schema and extraction       | Memory type mismatch, wrong  ||                      | store                    | prompts                     | namespace                    |+----------------------+--------------------------+-----------------------------+------------------------------+\n```\n\nThe last column is the one that matters. Every one of these can fail in a way that still produces a fluent, confident answer. JIN made a related point in a piece I read this week: an agent can find the document that says “Database: MySQL” and still give a wrong recommendation, because what it retrieved was the residue of a decision, not the reasoning behind it. Retrieval “worked.” The memory didn’t.\n\nSo if the failures are quiet, the evaluation has to be loud. That’s where the benchmarks come in, and where I started losing some faith in them.\n\nLoCoMo (Maharana et al., ACL 2024) is one of the most cited long-term conversational memory benchmarks. Long multi-session dialogues, a set of questions about them, and an answer key. If you’ve read a memory system launch post in the last year, there’s a good chance it had a LoCoMo number in it.\n\nEarlier this year a team at Penfield Labs published a systematic audit of that answer key. They went through all 1,540 non-adversarial questions and flagged 99 errors that actually change scores. That’s 6.4 percent. The categories are the ones that should make anyone building memory nervous: facts in the answer key that don’t exist in the source conversation, wrong temporal reasoning, and misattributed speakers.\n\nMy favorite example from the audit, because it’s so mundane: one expected answer names a specific sports car model. The conversation itself only ever says something like “this beauty,” and the image caption just describes a red sports car. The model name only exists in an internal annotation field, a search string annotators used to find stock photos, which no memory system ever ingests. A system that answers “a red sports car” is correct and gets marked wrong. A system that somehow says the exact model name is either very lucky or has seen the answer key.\n\nSix point four percent sounds small until you do the arithmetic. If 99 questions are wrong in the key, the honest ceiling on the benchmark is around 93.6 percent, not 100. Some published systems now report scores in the low 90s. At that range you’re not measuring memory anymore. You’re measuring how closely a system happens to agree with the mistakes.\n\nThen the judge. Modern LoCoMo evaluations don’t use exact-match scoring (which was too harsh, since “7 May 2023” and “May 7th, 2023” count as different). They use an LLM judge, typically gpt-4o-mini, comparing the system’s answer to the reference. The auditors generated deliberately wrong but topically adjacent answers for every question and ran them through the same judge setup used in published evaluations. The judge accepted about 63 percent of them.\n\nRead that one twice. Specific factual mistakes, like a wrong name or wrong date, got caught most of the time, around 89 percent. What slipped through were vague answers: responses that land on the right topic but don’t contain any of the specific details. And that is exactly what weak retrieval looks like. You find the right conversation, you pull nothing precise out of it, you produce a confident paraphrase of the general area. The benchmark doesn’t just fail to penalize that failure mode. It rewards it.\n\nIt gets worse when you stack the other findings from the same audit. There’s no standard pipeline, so each system uses its own ingestion, its own answer prompt, sometimes its own answer model, and then the numbers go into the same comparison table. Some category sample sizes are small enough that the auditors found more than half of adjacent comparisons within a category to be statistically indistinguishable. Independent people have reported trouble reproducing published results. And the often-suggested alternative, LongMemEval-S, has per-question haystacks small enough to fit in a modern context window, which makes it more of a long-context test than a memory test.\n\nI’ll admit my first reaction was a little smug, the “I knew leaderboards were fishy” kind. My second reaction was less comfortable. I had cited LoCoMo numbers myself, in a design doc, to justify picking one memory layer over another for a prototype. I never opened the dataset. I never looked at a single question. I trusted a number because it was in a table and the table was in a paper.\n\nWhen I left a comment on Rick’s piece asking whether anyone had re-run the popular systems against a corrected key, the honest answer turned out to be “partially.” The audit repo includes an adjusted-score analysis for several published systems, which is great work. But it can only correct the key it was given. It can’t tell you whether any of those systems will remember the things your users actually tell your agent.\n\nI don’t want to overcorrect into “benchmarks are useless.” LoCoMo is still useful for catching a system that’s badly broken. If something scores 30 percent, you learn something. The problem is resolution. Once systems cluster near the top, the noise in the key and the leniency of the judge are bigger than the differences you’re trying to see.\n\n```\n+-----------------------------------+----------------------------+-----------------------------+| Question you actually have        | Public benchmark score     | Your own golden set         |+-----------------------------------+----------------------------+-----------------------------+| Is this system fundamentally      | Yes, roughly               | Yes                         || broken?                           |                            |                             || Is A better than B by 3 points?   | No (inside the noise)      | Only for your data, but yes || Does it handle MY supersessions?  | No                         | Yes                         || Did the last upgrade regress it?  | No (nobody re-runs it)     | Yes, on every change        || Is a vague answer counted as a    | Often (lenient judge)      | No, if you use assertions   || pass?                             |                            |                             || Is the expected answer derivable  | Not always (6.4% errors)   | Only if you lint for it     || from what the system ingested?    |                            |                             |+-----------------------------------+----------------------------+-----------------------------+\n```\n\nThat last row became a rule in my own eval set, and I’ll show the code for it below.\n\nHere’s what I actually do now. It’s not glamorous, and it’s the same shape as the eval harness I described in my piece on prompts being consumables. The difference is that memory needs a different kind of assertion.\n\n**Start with 20 to 30 real interactions.** Not synthetic ones. Pull them from real sessions, real support tickets, real “why did the agent forget that” complaints. For Featpilot that meant things like a user stating a ticket format preference in one session and expecting it three sessions later, or a repo being renamed mid-project. If it never happened to a real user, it doesn’t go in.\n\n**Cover the failure shapes, not just the happy path.** My set is tagged by type, and I try to keep a few of each:\n\nThe funny thing is that three of those are exactly the categories where the LoCoMo key itself had errors. Those are the hard ones for humans to annotate too.\n\n**Use assertions, not a judge.** For retrieval-level memory checks you usually don’t need an LLM to grade anything. The question is simple: given what was written to memory, does the fact come back in the top k results when asked? That’s a yes or no, and it can be checked with a pattern match. No leniency, no “close enough, it mentioned the topic.” You can add a judged layer for the final generated answer later if you want, but the retrieval assertion is the foundation and it’s the part that catches silent regressions.\n\n**Lint the golden set itself.** Every expected answer must be derivable from the writes in that case. If the expected pattern doesn’t appear anywhere in what the system ingested, the case is wrong, not the system. That one rule would have caught the sports car problem.\n\n**Write down a policy for changing it.** This is the part people skip. Mine is short:\n\nHere’s a minimal, working version. It runs fully locally with Ollama for embeddings, with an optional Mem0 backend pointed at Ollama and a Qdrant container, so you can test a real memory system without paying for API calls.\n\nFirst, the local services:\n\n```\n# Local models for embeddings and (for Mem0) fact extractionollama pull nomic-embed-textollama pull llama3.1:8b# Vector store for the Mem0 backenddocker run -d --name qdrant -p 6333:6333 qdrant/qdrant# Python depspip install requests pytest mem0ai\n```\n\nThe golden set is plain JSON lines. Each case carries its own writes, so it runs in an isolated namespace and nothing bleeds between cases.\n\n```\n{\"id\": \"pref-ticket-format\", \"writes\": [\"For the payments repo, always use the Why / What / Acceptance Criteria layout in tickets.\"], \"query\": \"How should tickets for the payments repo be structured?\", \"expect_any\": [\"acceptance criteria\"], \"tags\": [\"recall\", \"preference\"], \"source\": \"session 2026-08-14\"}{\"id\": \"supersede-staging-db\", \"writes\": [\"Our staging database is MySQL 8.\", \"Update: we migrated staging to Postgres 16 last week and MySQL is gone.\"], \"query\": \"What database does staging run on right now?\", \"expect_any\": [\"postgres\"], \"forbid_top1\": [\"database is mysql\"], \"tags\": [\"supersession\", \"temporal\"], \"source\": \"incident notes 2026-07-02\"}{\"id\": \"speaker-billing-owner\", \"writes\": [\"The platform lead confirmed the billing service owns invoice numbering.\", \"A new engineer suggested the orders service should own invoice numbering eventually.\"], \"query\": \"Which service owns invoice numbering today?\", \"expect_any\": [\"billing service owns\"], \"forbid_top1\": [\"orders service should own\"], \"tags\": [\"speaker\"], \"source\": \"design review 2026-06-19\"}{\"id\": \"exact-config-key\", \"writes\": [\"The gateway timeout is controlled by PAYGW_TIMEOUT_MS, currently 30000.\"], \"query\": \"Which setting controls the payment gateway timeout?\", \"expect_any\": [\"PAYGW_TIMEOUT_MS\"], \"tags\": [\"identifier\"], \"source\": \"ticket FP-212\"}\n```\n\nSave that as golden_set.jsonl. Next, the adapters. Anything that can write, search and count can be plugged in.\n\n``` python\n# memory_adapters.pyfrom __future__ import annotationsimport mathimport osfrom typing import Protocolimport requestsOLLAMA_URL = os.getenv(\"OLLAMA_URL\", \"http://localhost:11434\")class MemoryAdapter(Protocol):    def write(self, namespace: str, text: str) -> None: ...    def search(self, namespace: str, query: str, k: int) -> list[str]: ...    def count(self, namespace: str) -> int: ...def embed(text: str, model: str = \"nomic-embed-text\") -> list[float]:    resp = requests.post(        f\"{OLLAMA_URL}/api/embed\",        json={\"model\": model, \"input\": text},        timeout=60,    )    resp.raise_for_status()    return resp.json()[\"embeddings\"][0]def _cosine(a: list[float], b: list[float]) -> float:    dot = sum(x * y for x, y in zip(a, b))    na = math.sqrt(sum(x * x for x in a))    nb = math.sqrt(sum(y * y for y in b))    return dot / (na * nb) if na and nb else 0.0class LocalVectorMemory:    \"\"\"Deliberately dumb baseline: store raw text, rank by cosine similarity.    No extraction, no conflict resolution. Every real system should beat this.\"\"\"    def __init__(self) -> None:        self._rows: dict[str, list[tuple[str, list[float]]]] = {}    def write(self, namespace: str, text: str) -> None:        self._rows.setdefault(namespace, []).append((text, embed(text)))    def search(self, namespace: str, query: str, k: int) -> list[str]:        q = embed(query)        rows = self._rows.get(namespace, [])        ranked = sorted(rows, key=lambda r: _cosine(q, r[1]), reverse=True)        return [text for text, _ in ranked[:k]]    def count(self, namespace: str) -> int:        return len(self._rows.get(namespace, []))MEM0_LOCAL_CONFIG = {    \"vector_store\": {        \"provider\": \"qdrant\",        \"config\": {\"host\": \"localhost\", \"port\": 6333, \"embedding_model_dims\": 768},    },    \"llm\": {        \"provider\": \"ollama\",        \"config\": {\"model\": \"llama3.1:8b\", \"ollama_base_url\": OLLAMA_URL},    },    \"embedder\": {        \"provider\": \"ollama\",        \"config\": {\"model\": \"nomic-embed-text\", \"ollama_base_url\": OLLAMA_URL},    },}def _items(result) -> list[dict]:    # Mem0 has returned both a bare list and {\"results\": [...]} across versions.    return result.get(\"results\", []) if isinstance(result, dict) else resultclass Mem0Adapter:    def __init__(self, config: dict | None = None) -> None:        from mem0 import Memory        self._m = Memory.from_config(config or MEM0_LOCAL_CONFIG)    def write(self, namespace: str, text: str) -> None:        self._m.add([{\"role\": \"user\", \"content\": text}], user_id=namespace)    def search(self, namespace: str, query: str, k: int) -> list[str]:        return [i[\"memory\"] for i in _items(self._m.search(query, user_id=namespace, limit=k))]    def count(self, namespace: str) -> int:        return len(_items(self._m.get_all(user_id=namespace)))\n```\n\nNow the check itself, including the lint rule for the golden set:\n\n``` python\n# recall_check.pyfrom __future__ import annotationsimport jsonimport reimport uuidfrom dataclasses import dataclass, field@dataclassclass RecallCase:    id: str    writes: list[str]    query: str    expect_any: list[str]    forbid_top1: list[str] = field(default_factory=list)    k: int = 5    tags: list[str] = field(default_factory=list)    source: str = \"\"@dataclassclass RecallResult:    case_id: str    passed: bool    reason: str    hits: list[str]def load_cases(path: str) -> list[RecallCase]:    with open(path, encoding=\"utf-8\") as f:        return [RecallCase(**json.loads(line)) for line in f if line.strip()]def lint_case(case: RecallCase) -> list[str]:    \"\"\"The golden set gets tested too. An expected answer that isn't in    anything the system ingested is an annotation error, not a memory failure.\"\"\"    problems = []    corpus = \"\\n\".join(case.writes)    for pattern in case.expect_any:        if not re.search(pattern, corpus, re.IGNORECASE):            problems.append(f\"expected pattern {pattern!r} not derivable from writes\")    if not case.writes:        problems.append(\"case has no writes\")    return problemsdef check_recall(adapter, case: RecallCase) -> RecallResult:    ns = f\"eval-{case.id}-{uuid.uuid4().hex[:8]}\"    for text in case.writes:        adapter.write(ns, text)    if adapter.count(ns) == 0:        return RecallResult(case.id, False, \"nothing was stored\", [])    hits = adapter.search(ns, case.query, case.k)    if not hits:        return RecallResult(case.id, False, \"search returned nothing\", hits)    joined = \"\\n\".join(hits)    if not any(re.search(p, joined, re.IGNORECASE) for p in case.expect_any):        return RecallResult(case.id, False, f\"none of {case.expect_any} in top {case.k}\", hits)    for pattern in case.forbid_top1:        if re.search(pattern, hits[0], re.IGNORECASE):            return RecallResult(case.id, False, f\"stale or misattributed fact ranked first: {pattern!r}\", hits)    return RecallResult(case.id, True, \"ok\", hits)\n```\n\nAnd the pytest file that ties it together:\n\n``` python\n# test_memory_recall.pyimport osimport pytestfrom memory_adapters import LocalVectorMemory, Mem0Adapterfrom recall_check import check_recall, lint_case, load_casesCASES = load_cases(os.getenv(\"GOLDEN_SET\", \"golden_set.jsonl\"))IDS = [c.id for c in CASES]@pytest.fixture(scope=\"session\")def adapter():    if os.getenv(\"MEMORY_BACKEND\", \"local\") == \"mem0\":        return Mem0Adapter()    return LocalVectorMemory()@pytest.mark.parametrize(\"case\", CASES, ids=IDS)def test_golden_case_is_well_formed(case):    problems = lint_case(case)    assert not problems, f\"{case.id}: {problems}\"@pytest.mark.parametrize(\"case\", CASES, ids=IDS)def test_memory_recalls_fact(adapter, case):    result = check_recall(adapter, case)    assert result.passed, (        f\"{result.reason}\\n\"        f\"tags: {case.tags}  source: {case.source}\\n\"        f\"top hits: {result.hits}\"    )\n```\n\nRun the baseline, then the real system:\n\n```\npytest -q test_memory_recall.pyMEMORY_BACKEND=mem0 pytest -q test_memory_recall.py\n```\n\nThe baseline run is important and I almost skipped it. The dumb cosine store is what every memory system is supposed to beat. In my experience it does fine on plain recall and exact identifiers when the set is tiny, and it falls over on supersession, because both the old and the new fact sit in the store and nothing tells it which one is current. If your fancy memory layer doesn’t clearly beat the baseline on the supersession and speaker tags, you just learned something no leaderboard would have told you.\n\nOne more small check, because of how this whole article started. Silent index shrinkage is its own failure mode, and it’s cheap to guard against:\n\n```\n# index_canary.pydef check_index_health(expected_docs: int, bm25_docs: int, vector_docs: int, tolerance: float = 0.02) -> list[str]:    \"\"\"Record expected_docs at re-index time. Run this after every re-index and on a schedule.    Fusion will happily hide an empty list; a count comparison will not.\"\"\"    problems = []    floor = expected_docs * (1 - tolerance)    if bm25_docs < floor:        problems.append(f\"keyword index has {bm25_docs}, expected about {expected_docs}\")    if vector_docs < floor:        problems.append(f\"vector index has {vector_docs}, expected about {expected_docs}\")    if abs(bm25_docs - vector_docs) > expected_docs * tolerance:        problems.append(f\"indexes disagree: bm25={bm25_docs} vector={vector_docs}\")    return problems\n```\n\nThat function is ten lines and would have saved me four days. I also added one exact-identifier case per repo to the golden set, queried against the keyword side alone, so a dead BM25 index fails loudly instead of being papered over by embeddings.\n\nA couple of weeks ago I argued that prompts are consumables and evals are capital. Prompts encode how a specific model at a specific version responds to specific phrasing, so they decay with every upgrade. Tests encode what correct behavior looks like in your domain, which changes far more slowly, so they compound.\n\nMemory systems make that argument sharper, because there are more moving layers underneath you. The extraction model inside Mem0 will change. Graphiti’s entity resolution will get better, and occasionally worse. Letta will change how its agent decides what to page in. Anthropic will change how the model decides what to write into its memory files. Your embedding model will get swapped for a cheaper one by someone optimizing costs. Every one of those changes can move your memory quality in either direction, silently, while the API keeps returning 200.\n\nA benchmark number is a snapshot of someone else’s system, on someone else’s data, graded by someone else’s judge against an answer key that, as it turns out, was wrong about one question in sixteen. You cite it once and it never updates. It can’t tell you what happened after your last dependency bump.\n\nThirty honest cases in a JSON lines file can. They’re specific to what your users actually say. They’re graded by assertions that don’t get tired or lenient. They’re linted so the key can’t quietly drift away from what the system could possibly know. And they run on every change, against whatever memory layer you’re using this quarter, which is the only kind of number I actually trust now.\n\nI’m still going to read memory papers. I still find them useful for ideas, especially the temporal graph work. But when a launch post shows me a LoCoMo score in the low 90s, I no longer read it as “this system remembers 91 percent of things.” I read it as “this system agreed with a flawed key, under a lenient judge, on a pipeline nobody else can reproduce, 91 percent of the time.” Then I go add it as a backend to my own test file and see what it does with my supersession cases.\n\nThat’s usually a more interesting afternoon than reading the leaderboard.\n\n**Tags:** Agent Memory, AI Agents, LLM Evaluation, Retrieval Augmented Generation, LLM Engineering, Benchmarks, Machine Learning, Python, Mem0, Ollama\n\n[What a 6.4 Percent Error Rate in a Benchmark’s Answer Key Taught Me About Trusting Agent Memory…](https://pub.towardsai.net/what-a-6-4-percent-error-rate-in-a-benchmarks-answer-key-taught-me-about-trusting-agent-memory-58415a8a8842) was originally published in [Towards AI](https://pub.towardsai.net) on Medium, where people are continuing the conversation by highlighting and responding to this story.", "url": "https://wpnews.pro/news/what-a-6-4-percent-error-rate-in-a-benchmarks-answer-key-taught-me-about-agent", "canonical_source": "https://pub.towardsai.net/what-a-6-4-percent-error-rate-in-a-benchmarks-answer-key-taught-me-about-trusting-agent-memory-58415a8a8842?source=rss----98111c9905da---4", "published_at": "2026-10-16 20:01:01+00:00", "updated_at": "2026-10-16 20:20:49.733032+00:00", "lang": "en", "topics": ["ai-agents", "ai-tools", "ai-infrastructure", "mlops", "artificial-intelligence"], "entities": ["Featpilot", "Rick Hightower", "Mem0", "Zep", "Graphiti", "MemGPT", "Letta", "BM25"], "also_reported_by": [], "alternates": {"html": "https://wpnews.pro/news/what-a-6-4-percent-error-rate-in-a-benchmarks-answer-key-taught-me-about-agent", "markdown": "https://wpnews.pro/news/what-a-6-4-percent-error-rate-in-a-benchmarks-answer-key-taught-me-about-agent.md", "text": "https://wpnews.pro/news/what-a-6-4-percent-error-rate-in-a-benchmarks-answer-key-taught-me-about-agent.txt", "jsonld": "https://wpnews.pro/news/what-a-6-4-percent-error-rate-in-a-benchmarks-answer-key-taught-me-about-agent.jsonld"}}