Memory layers for AI agents love to quote LongMemEval. You will see 94%, 95%, 96%. Those numbers are real, but they usually measure retrieval: was one of the right sessions in the top 5 results? (recall_any@5). LongMemEval's own metric is different: did the model answer the question correctly?, graded by a GPT-4o judge.
I wanted to know how far apart those two numbers are, so I measured both on the same questions.
agentmemory reports 95.2% R@5 on LongMemEval_S, and to its credit it says plainly that this is retrieval recall and not QA accuracy. It also publishes the sessions its hybrid search (BM25 + embeddings) returned for each of the 500 questions.
So I took exactly those results, unchanged, pinned by commit and hash:
Then I did the same with Graph-MIND, the memory layer I build, whose shipped retrieval returns a ~3.4k-token packet of individual turns.
Only the retrieval differs. Same questions, same reader, same judge.
| answer accuracy (499 Qs) | never used for tuning (379) | text the reader sees per question | |
|---|---|---|---|
| agentmemory (its top 5 sessions) | 80.4% | 78.6% | ~16.5k tokens |
| Graph-MIND | 89.0% | 87.1% | ~3.4k tokens |
Graph-MIND alone was right on 64 questions and agentmemory alone on 21; exact McNemar p = 3.3e-6.
By question type (Graph-MIND / agentmemory): Three things happen between "a right session is in the top 5" and "the answer is right":
None of this makes agentmemory bad; 80% on LongMemEval with a small reader is respectable, and whole sessions are its natural unit. The point is that R@5 and QA accuracy are different measurements, and comparing one project's R@5 with another's QA (or with each other's R@5 on different units: sessions vs. turns vs. chunks) does not tell you which one answers better.
Everything is reproducible: the script is research/rival_agentmemory.py, and every question, answer and judge verdict is in research/runs/graph-v9.6-rival-agentmemory-k5. If the method is wrong, I would like to know.
A local memory shared by the coding agents in your terminal (Claude Code, Codex, Cursor, Gemini CLI, Copilot CLI, OpenCode). It stores every turn verbatim in SQLite on your machine, with no model call at write time, and retrieves with word search plus a small local embedding model. Open source (AGPL), one-line install.
The takeaway, whichever memory you use: ask for answer accuracy, on the same reader and judge.