Follow-up to the Muninn cliffhanger. Part of The Organism Files.
Yesterday I told you my AI partner and I built a memory system — Muninn — overnight and entered it in a public leaderboard against Tencent, Mem0, Cognee, and MemOS, and that I'd post the score either way.
Here's the either-way: there's no score, because we were late. That board is the Agent Memory Leaderboard, and its submission window had already closed before we entered. Rookie miss — I own it. The next cycle opens in September, and we'll be early this time.
But "late" doesn't mean "nothing." Here's what we built, what it scored when we ran it ourselves, and — because there's more than one iron in this fire — a result on a second, separate benchmark that isn't self-judged.
The Agent Memory Leaderboard hands your system two jobs: Add (they give you conversation history) and Search (they give you a question, you return the memories that answer it). Their platform fixes the answer model — gpt-4o-mini — so the only variable is your memory. Muninn is a hybrid retriever: keyword + semantic embeddings + a query-expansion pass, fused, with a check-in/check-out rule where a fact restated later supersedes the older copy.
Since we missed the official window, we ran their own public pipeline ourselves, on the LoCoMo dataset. Their grading model isn't published, so we used gpt-4o-mini as a stand-in judge too — which means every number here is an internal estimate, not an official score. With that caveat loud: our best config landed at 72.9% (≈66% across the full set). It's our yardstick, not theirs — but it's a real measurement of what we'd have submitted.
Same memory engine, second board. On the agent-memory leaderboard we call it Muninn; on ** LongMemEval-V2** — a separate benchmark, and one we
| System | Accuracy | Latency/query |
|---|---|---|
| RAG + notes (baseline) | 51.0% | ~0.2s |
| Perpetual Recall — our submission | ||
| 56.98% | ||
| ~2.3s | ||
| ↳ same system, zero reader-directed text | ||
| 54.32% | ||
| ~2.8s | ||
| AgentRunbook-R | 58.6% | 26.9s |
| Codex (vanilla) | 69.9% | 177.2s |
| AgentRunbook-C (best) | 74.9% | 108.3s |
(Those competitor rows are the benchmark's own published baselines. The public leaderboard still reads "entries coming soon" — so 56.98% is our submitted number, filed and awaiting the maintainer's review, not a confirmed rank.)
On raw accuracy we're mid-pack. But look at the seconds: every system that outscores us takes 27 to 180 seconds a query. We answer in under three. Near-frontier accuracy at a fraction of the time — and even stripped to strict-provenance, with zero text aimed at the reader, we're still at 54.32%, above the RAG baseline. That floor is the number I'll stand on.
Here's the one I'm almost prouder of, and it's deliberately not in that table: a configuration that scored 65.41% — a dozen points over our clean run, enough to sit third on that board, behind only the two heaviest agent systems that take a hundred-plus seconds a query. We answer in under three. I didn't submit it. On purpose.
The jump comes from a grounding header — a fixed block of instructions we prepend to the evidence before the reader sees it: ground your answer only in what's recorded; if the item asked for isn't here, say it doesn't exist; if the question asks for "the 4th tab" and only three are recorded, that's a false premise. Identical text on every question. And almost every one of the ~50 questions it gained was a trap — an abstention or a false-premise question the reader would otherwise have confidently made up an answer to.
Which is exactly why I held it back. That header doesn't make the memory better — it coaches the reader on how to answer. On a benchmark built to isolate the memory, I couldn't certify that as fair to the systems we'd be ranked against. So we sealed it, disclosed the exact instruction text, and offered it for an explicit scope ruling instead of putting the number on the board. It was too good to trust — so we didn't.
And here's the part I genuinely can't answer — a hypothesis, not a claim. In a real product that header isn't a cheat, it's the point: ship it blank, fill it in per business, and you've got a coded scaffold that makes the AI ground itself correctly on the actual job. It solves legitimately in the field. There's even a signal it's the field's winning move, not a trick — the top system on that board earns its edge from a structured "manifest" of exactly this shape. But a benchmark that has to strip the scaffold out to stay fair may never credit the thing that does the most work in production. Where's the line between a grounding scaffold and coaching the grader — and can a fair test ever tell them apart? If you've thought about this, or there's prior work I've missed, I want to hear it.
September's the real shot at the board I missed.
Until then — I want the argument. Where does the grounding-header line actually fall, scaffold or coaching? Is the clean 54% the only number that counts, or is the withheld 65% the one that matters? Is there prior work here I've missed? Drop it in the comments. And if any of this is worth a closer look — the configs, the disclosures, the exact runs — tell me and I'll go as deep as you want to go. Building in the open only means something if you get to check my work.
Part of The Organism Files — building a verification-first AI partnership from scratch, out loud.