{"slug": "i-missed-the-deadline-so-i-ran-the-benchmark-myself-to-prep-for-its-reopening", "title": "I missed the deadline — so I ran the benchmark myself to prep for its reopening", "summary": "A developer who built the Muninn memory system with an AI partner missed the submission window for the Agent Memory Leaderboard, so they ran the benchmark's public pipeline themselves and reported an internal estimate of 72.9% accuracy. On the separate LongMemEval-V2 benchmark, their submitted system scored 56.98% accuracy with under three seconds per query, outperforming heavier systems in speed. The developer deliberately withheld a configuration that scored 65.41% because it included a grounding header that coached the reader model, which they deemed unfair for a memory-focused benchmark.", "body_md": "*Follow-up to the Muninn cliffhanger. Part of The Organism Files.*\n\nYesterday I told you my AI partner and I built a memory system — **Muninn** — overnight and entered it in a public leaderboard against Tencent, Mem0, Cognee, and MemOS, and that I'd post the score either way.\n\nHere's the either-way: **there's no score, because we were late.** That board is the Agent Memory Leaderboard, and its submission window had already closed before we entered. Rookie miss — I own it. The next cycle opens in September, and we'll be early this time.\n\nBut \"late\" doesn't mean \"nothing.\" Here's what we built, what it scored when we ran it ourselves, and — because there's more than one iron in this fire — a result on a *second, separate* benchmark that isn't self-judged.\n\nThe Agent Memory Leaderboard hands your system two jobs: **Add** (they give you conversation history) and **Search** (they give you a question, you return the memories that answer it). Their platform fixes the answer model — **gpt-4o-mini** — so the only variable is your memory. Muninn is a hybrid retriever: keyword + semantic embeddings + a query-expansion pass, fused, with a check-in/check-out rule where a fact restated later supersedes the older copy.\n\nSince we missed the official window, we ran their *own public pipeline* ourselves, on the LoCoMo dataset. Their grading model isn't published, so we used gpt-4o-mini as a **stand-in judge** too — which means every number here is an **internal estimate, not an official score.** With that caveat loud: our best config landed at **72.9%** (≈66% across the full set). It's our yardstick, not theirs — but it's a real measurement of what we'd have submitted.\n\nSame memory engine, second board. On the agent-memory leaderboard we call it **Muninn**; on ** LongMemEval-V2** — a separate benchmark, and one we\n\n| System | Accuracy | Latency/query |\n|---|---|---|\n| RAG + notes (baseline) | 51.0% | ~0.2s |\nPerpetual Recall — our submission |\n56.98% |\n~2.3s |\n↳ same system, zero reader-directed text\n|\n54.32% |\n~2.8s |\n| AgentRunbook-R | 58.6% | 26.9s |\n| Codex (vanilla) | 69.9% | 177.2s |\n| AgentRunbook-C (best) | 74.9% | 108.3s |\n\n*(Those competitor rows are the benchmark's own published baselines. The public leaderboard still reads \"entries coming soon\" — so 56.98% is our* submitted *number, filed and awaiting the maintainer's review, not a confirmed rank.)*\n\nOn raw accuracy we're mid-pack. But look at the **seconds**: every system that outscores us takes **27 to 180 seconds a query.** We answer in under three. Near-frontier accuracy at a fraction of the time — and even stripped to strict-provenance, with *zero* text aimed at the reader, we're still at 54.32%, above the RAG baseline. **That floor is the number I'll stand on.**\n\nHere's the one I'm almost prouder of, and it's deliberately *not* in that table: a configuration that scored **65.41%** — a dozen points over our clean run, enough to sit third on that board, behind only the two heaviest agent systems that take a hundred-plus seconds a query. We answer in under three. **I didn't submit it. On purpose.**\n\nThe jump comes from a *grounding header* — a fixed block of instructions we prepend to the evidence before the reader sees it: *ground your answer only in what's recorded; if the item asked for isn't here, say it doesn't exist; if the question asks for \"the 4th tab\" and only three are recorded, that's a false premise.* Identical text on every question. And almost every one of the ~50 questions it gained was a **trap** — an abstention or a false-premise question the reader would otherwise have confidently made up an answer to.\n\nWhich is exactly why I held it back. That header doesn't make the *memory* better — it coaches the *reader* on how to answer. On a benchmark built to isolate the memory, I couldn't certify that as fair to the systems we'd be ranked against. So we sealed it, disclosed the exact instruction text, and offered it for an explicit scope ruling instead of putting the number on the board. **It was too good to trust — so we didn't.**\n\nAnd here's the part I genuinely can't answer — a hypothesis, not a claim. In a real product that header isn't a cheat, it's the *point*: ship it blank, fill it in per business, and you've got a coded scaffold that makes the AI ground itself correctly on the actual job. It solves legitimately in the field. There's even a signal it's the field's *winning* move, not a trick — the top system on that board earns its edge from a structured \"manifest\" of exactly this shape. But a benchmark that has to strip the scaffold out to stay fair may never credit the thing that does the most work in production. **Where's the line between a grounding scaffold and coaching the grader — and can a fair test ever tell them apart?** If you've thought about this, or there's prior work I've missed, I want to hear it.\n\nSeptember's the real shot at the board I missed.\n\nUntil then — **I want the argument.** Where does the grounding-header line actually fall, scaffold or coaching? Is the clean 54% the only number that counts, or is the withheld 65% the one that matters? Is there prior work here I've missed? Drop it in the comments. And if any of this is worth a closer look — the configs, the disclosures, the exact runs — tell me and I'll go as deep as you want to go. Building in the open only means something if you get to check my work.\n\n*Part of The Organism Files — building a verification-first AI partnership from scratch, out loud.*", "url": "https://wpnews.pro/news/i-missed-the-deadline-so-i-ran-the-benchmark-myself-to-prep-for-its-reopening", "canonical_source": "https://dev.to/bryanw/i-missed-the-deadline-so-i-ran-the-benchmark-myself-to-prep-for-its-reopening-774", "published_at": "2026-08-24 21:43:40+00:00", "updated_at": "2026-08-24 22:13:37.195939+00:00", "lang": "en", "topics": ["artificial-intelligence", "machine-learning", "large-language-models", "ai-research", "ai-tools"], "entities": ["Muninn", "Tencent", "Mem0", "Cognee", "MemOS", "Agent Memory Leaderboard", "LongMemEval-V2", "gpt-4o-mini"], "alternates": {"html": "https://wpnews.pro/news/i-missed-the-deadline-so-i-ran-the-benchmark-myself-to-prep-for-its-reopening", "markdown": "https://wpnews.pro/news/i-missed-the-deadline-so-i-ran-the-benchmark-myself-to-prep-for-its-reopening.md", "text": "https://wpnews.pro/news/i-missed-the-deadline-so-i-ran-the-benchmark-myself-to-prep-for-its-reopening.txt", "jsonld": "https://wpnews.pro/news/i-missed-the-deadline-so-i-ran-the-benchmark-myself-to-prep-for-its-reopening.jsonld"}}