{"slug": "95-recall-80-answers-what-longmemeval-retrieval-scores-don-t-tell-you", "title": "95% recall, 80% answers: what LongMemEval retrieval scores don't tell you", "summary": "A developer measured answer accuracy against retrieval recall on LongMemEval, finding that agentmemory's reported 95.2% R@5 retrieval score translated to 80.4% answer accuracy across 499 questions, while the developer's own Graph-MIND memory layer reached 89.0% accuracy using roughly 3.4k tokens per question versus agentmemory's ~16.5k. On the 379 questions never used for tuning, Graph-MIND scored 87.1% versus 78.6% for agentmemory, with an exact McNemar p of 3.3e-6. The developer published the evaluation script and full run data, arguing that R@5 and QA accuracy are distinct measurements that should not be compared across projects.", "body_md": "Memory layers for AI agents love to quote LongMemEval. You will see 94%, 95%, 96%. Those numbers are real, but they usually measure retrieval: *was one of the right sessions in the top 5 results?* (recall_any@5). LongMemEval's own metric is different: *did the model answer the question correctly?*, graded by a GPT-4o judge.\n\nI wanted to know how far apart those two numbers are, so I measured both on the same questions.\n\n[agentmemory](https://github.com/rohitg00/agentmemory) reports **95.2% R@5** on LongMemEval_S, and to its credit it says plainly that this is retrieval recall and not QA accuracy. It also publishes the sessions its hybrid search (BM25 + embeddings) returned for each of the 500 questions.\n\nSo I took exactly those results, unchanged, pinned by commit and hash:\n\nThen I did the same with [Graph-MIND](https://github.com/goyohan0611-png/graph-mind), the memory layer I build, whose shipped retrieval returns a ~3.4k-token packet of individual turns.\n\nOnly the retrieval differs. Same questions, same reader, same judge.\n\n|  | answer accuracy (499 Qs) | never used for tuning (379) | text the reader sees per question | \n|---|---|---|---|\n| agentmemory (its top 5 sessions) | 80.4% | 78.6% | ~16.5k tokens | \n| Graph-MIND | 89.0% | 87.1% | ~3.4k tokens | \n\nGraph-MIND alone was right on 64 questions and agentmemory alone on 21; exact McNemar p = 3.3e-6.\n\nBy question type (Graph-MIND / agentmemory):\n\nThree things happen between \"a right session is in the top 5\" and \"the answer is right\":\n\nNone of this makes agentmemory bad; 80% on LongMemEval with a small reader is respectable, and whole sessions are its natural unit. The point is that **R@5 and QA accuracy are different measurements**, and comparing one project's R@5 with another's QA (or with each other's R@5 on different units: sessions vs. turns vs. chunks) does not tell you which one answers better.\n\nEverything is reproducible: the script is [`research/rival_agentmemory.py`](https://github.com/goyohan0611-png/graph-mind/blob/main/research/rival_agentmemory.py), and every question, answer and judge verdict is in [`research/runs/graph-v9.6-rival-agentmemory-k5`](https://github.com/goyohan0611-png/graph-mind/tree/main/research/runs/graph-v9.6-rival-agentmemory-k5). If the method is wrong, I would like to know.\n\nA local memory shared by the coding agents in your terminal (Claude Code, Codex, Cursor, Gemini CLI, Copilot CLI, OpenCode). It stores every turn verbatim in SQLite on your machine, with no model call at write time, and retrieves with word search plus a small local embedding model. Open source (AGPL), one-line install.\n\n**The takeaway, whichever memory you use: ask for answer accuracy, on the same reader and judge.**", "url": "https://wpnews.pro/news/95-recall-80-answers-what-longmemeval-retrieval-scores-don-t-tell-you", "canonical_source": "https://dev.to/goyohan0611png/95-recall-80-answers-what-longmemeval-retrieval-scores-dont-tell-you-5dg4", "published_at": "2026-10-19 02:35:25+00:00", "updated_at": "2026-10-19 02:48:55.228386+00:00", "lang": "en", "topics": ["ai-agents", "ai-research", "large-language-models", "ai-tools"], "entities": ["LongMemEval", "agentmemory", "Graph-MIND", "GPT-4o"], "also_reported_by": [], "alternates": {"html": "https://wpnews.pro/news/95-recall-80-answers-what-longmemeval-retrieval-scores-don-t-tell-you", "markdown": "https://wpnews.pro/news/95-recall-80-answers-what-longmemeval-retrieval-scores-don-t-tell-you.md", "text": "https://wpnews.pro/news/95-recall-80-answers-what-longmemeval-retrieval-scores-don-t-tell-you.txt", "jsonld": "https://wpnews.pro/news/95-recall-80-answers-what-longmemeval-retrieval-scores-don-t-tell-you.jsonld"}}