95% recall, 80% answers: what LongMemEval retrieval scores don't tell you

A developer measured answer accuracy against retrieval recall on LongMemEval, finding that agentmemory's reported 95.2% R@5 retrieval score translated to 80.4% answer accuracy across 499 questions, while the developer's own Graph-MIND memory layer reached 89.0% accuracy using roughly 3.4k tokens per question versus agentmemory's ~16.5k. On the 379 questions never used for tuning, Graph-MIND scored 87.1% versus 78.6% for agentmemory, with an exact McNemar p of 3.3e-6. The developer published the evaluation script and full run data, arguing that R@5 and QA accuracy are distinct measurements that should not be compared across projects.

Memory layers for AI agents love to quote LongMemEval. You will see 94%, 95%, 96%. Those numbers are real, but they usually measure retrieval: was one of the right sessions in the top 5 results? recall any@5 . LongMemEval's own metric is different: did the model answer the question correctly? , graded by a GPT-4o judge. I wanted to know how far apart those two numbers are, so I measured both on the same questions. agentmemory https://github.com/rohitg00/agentmemory reports 95.2% R@5 on LongMemEval S, and to its credit it says plainly that this is retrieval recall and not QA accuracy. It also publishes the sessions its hybrid search BM25 + embeddings returned for each of the 500 questions. So I took exactly those results, unchanged, pinned by commit and hash: Then I did the same with Graph-MIND https://github.com/goyohan0611-png/graph-mind , the memory layer I build, whose shipped retrieval returns a ~3.4k-token packet of individual turns. Only the retrieval differs. Same questions, same reader, same judge. | | answer accuracy 499 Qs | never used for tuning 379 | text the reader sees per question | |---|---|---|---| | agentmemory its top 5 sessions | 80.4% | 78.6% | ~16.5k tokens | | Graph-MIND | 89.0% | 87.1% | ~3.4k tokens | Graph-MIND alone was right on 64 questions and agentmemory alone on 21; exact McNemar p = 3.3e-6. By question type Graph-MIND / agentmemory : Three things happen between "a right session is in the top 5" and "the answer is right": None of this makes agentmemory bad; 80% on LongMemEval with a small reader is respectable, and whole sessions are its natural unit. The point is that R@5 and QA accuracy are different measurements , and comparing one project's R@5 with another's QA or with each other's R@5 on different units: sessions vs. turns vs. chunks does not tell you which one answers better. Everything is reproducible: the script is research/rival agentmemory.py https://github.com/goyohan0611-png/graph-mind/blob/main/research/rival agentmemory.py , and every question, answer and judge verdict is in research/runs/graph-v9.6-rival-agentmemory-k5 https://github.com/goyohan0611-png/graph-mind/tree/main/research/runs/graph-v9.6-rival-agentmemory-k5 . If the method is wrong, I would like to know. A local memory shared by the coding agents in your terminal Claude Code, Codex, Cursor, Gemini CLI, Copilot CLI, OpenCode . It stores every turn verbatim in SQLite on your machine, with no model call at write time, and retrieves with word search plus a small local embedding model. Open source AGPL , one-line install. The takeaway, whichever memory you use: ask for answer accuracy, on the same reader and judge.