How we measured a markdown knowledge graph as agent memory — lost to grep, rebuilt our search and editing primitives, retracted our own best number, and ended with a $4.50 curator whose store reads back at 96% of a hand-built ceiling in a single call.
Every AI agent product eventually hits the same wall: the model forgets. Context windows end, sessions restart, and everything the agent learned about you evaporates. An industry has grown up around this — hosted "memory layers" that ingest your conversations, extract facts with an LLM, embed them into a vector store, and sell retrieval back to you by the API call.
IWE starts from a different premise. It is a local-first knowledge tool: your notes are plain markdown files in a directory, connected by links into a graph, readable by any editor and greppable by any tool. If that graph is a good memory substrate, an agent should be able to write its memories as linked markdown pages and read them back with graph-aware tools — no embeddings, no hosted pipeline, no per-retrieval meter. Memories you can open in your editor and correct by hand.
But "should be able to" is a hypothesis, not a feature. So we built a benchmark to test it — and the honest version of this story is that the benchmark spent most of its life telling us our hypothesis was wrong, each time pointing at exactly what to fix. This article is the story of that loop: measure, lose, build, measure again. By the end, the benchmark had reshaped IWE's search engine, motivated an entire block-level editing language, killed several of our own design rules, retracted its own most exciting number, and settled somewhere we did not predict: the cheapest curation model we could run, kept honest by mechanical guards rather than instructions, writing a store that a single retrieval call reads back at 96 percent of a hand-built ceiling — while the humblest baseline in the study, grep over the raw transcripts, still refuses to be beaten on accuracy alone. Both halves of that ending are the point.
We wanted numbers comparable to the published agent-memory literature, so we adopted its standard instrument. If you haven't met it before, here is everything you need.
The corpus. LOCOMO (Maharana et al., 2024) is the dataset behind most published memory-system evaluations, including Mem0's. It is a set of ten very long fictional conversations, each a months-long text-message relationship between two people — think two friends catching up every week or two. A conversation is split into 19 to 35 sessions, each stamped with a date and time ("1:56 pm on 8 May, 2023"), and the pairs live ordinary lives across it: one loses a job and opens a dance studio, the other launches a clothing store; someone paints, someone runs a charity race, kids get taken to museums. A full conversation runs to hundreds of turns over half a year of simulated time. The two development conversations this article iterates on follow Caroline and Melanie, and Jon and Gina (nineteen sessions, January to July 2023).
The task. After the entire conversation has been ingested — every session, in chronological order — the system answers a battery of questions about it, between roughly 80 and 200 per conversation. Questions arrive one at a time, with no conversation context attached: whatever the system remembers is all it has. They come in four scored flavors:
A fifth category of adversarial, deliberately unanswerable questions exists in the dataset; we exclude it, following the methodology behind every published number we compare against. That leaves 1,540 scoreable questions across the ten conversations.
The metric. Answers are graded by an LLM judge that sees three things — the question, the dataset's gold answer, and the system's answer — and outputs one word: CORRECT or WRONG. No partial credit. We use the judge prompt from Mem0's published evaluation harness verbatim, so our grading is the same grading behind the numbers we compare to. J is the fraction judged correct: J = 0.85 means 17 of 20. The judge is strict to a fault — "the week of May 29" against a gold of "the week before June 9" is WRONG, full stop — and the answer key has known defects (every failure audit we ran surfaced golds that are wrong or narrower than the conversation supports). Both problems cut against every system equally, which is the point: comparability, not perfection. We also track token-overlap F1 as a secondary signal, but it rewards phrasing rather than truth, so J is the headline everywhere below.
The anchors. Published results to keep in mind while reading ours: a plain filesystem-plus-grep baseline scores about 0.74 (Letta's rerun), stuffing the entire conversation into the context window scores about 0.73, and memory products cluster between the high 0.60s and the mid 0.70s under strict judges. (Vendors' own reruns with lenient judges produce higher numbers; we ignore those.)
Our harness has one unusual property: everything runs through claude -p — the answering agents, the curation agent, and the judge. Unless stated otherwise, Sonnet fills every one of those roles; the cast changes twice in this story — the curator drops to a cheaper model midway, and the frozen test configuration pins answering and judging to
claude-sonnet-4-6
— and both changes are flagged where they happen. No direct API calls, no Python evaluation stack; the whole thing is a Rust cargo xtask
in a small repo. Each measurement spawns a headless Claude Code session in an isolated workspace: a dedicated configuration profile so no personal CLAUDE.md
or settings leak in, an explicit tool allowlist per experimental arm, --strict-mcp-config
so only the workspace's own MCP server is visible, and a budget cap per question. We verified the isolation empirically — sentinel instructions planted in ancestor directories, probes confirming they can't reach the agent — because a memory benchmark contaminated by the operator's own notes would be worthless.Before trusting the judge, we calibrated it: the full-context arm (whole transcript inline, no tools) scored J = 0.75 on our pilot conversation against the published 0.73 for the same configuration. Close enough to compare directionally. (The stricter frozen judge re-scores those same answers at 0.70; the closing section returns to judges at length.)
Two validity rules mattered more than anything else in the design:
The curator is question-blind. The agent that builds the memory never sees the benchmark questions — it ingests sessions one at a time, in order, with no lookahead, exactly as a real memory system would. The test for every prompt revision was: would this instruction make sense for any personal-conversation corpus? If it only made sense for LOCOMO, it was overfitting and it didn't ship.
The test set stays sealed — mostly. We iterate on two development conversations; the other eight are reserved for frozen configurations. The early rounds below are n=20 dev slices, single repetition — differences of one question (0.05 J) are within noise, and the slices skew temporal, so that category carries much of J. Treat them as directional. The methods notes at the end give the full accounting of how the test set was actually spent; the check that matters is that the final sealed-first-contact numbers match the iterated ones.
The experimental design is an ablation ladder. Each arm gives the answering agent the same question and the same budget, and varies only what memory it has and what tools it may use:
| arm | memory | tools |
|---|---|---|
full-context |
||
| the raw transcript, inline | none | |
fs |
||
| raw transcript files | Grep / Glob / Read | |
curated |
||
| notes only — an agent-built knowledge graph | ||
| IWE graph tools | ||
curated-fs |
||
| the same notes | Grep / Glob / Read only |
The curated
arms are the thesis: an agent replays the conversation session by session and distills it into an IWE graph — entity pages, dated event pages with machine-readable frontmatter, category pages, everything cross-linked. The raw transcript is then deleted from the workspace. What the curator failed to write down is gone, exactly like memory. The curated-fs
control is the most important line in the table: same notes, dumbest possible retrieval. It separates "the notes are good" from "the tools are good" — and it will embarrass us more than once.
The first full run (development conversation one, twenty questions, curation cost $19.15) came back:
| arm | J | cost/20q | turns |
|---|---|---|---|
full-context (whole transcript inline) |
|||
| 0.75 | $4.05 | 1.0 | |
fs (grep the raw transcript) |
|||
| 0.75 | $1.17 | 4.0 | |
curated (our thesis) |
|||
| 0.70 | |||
| $5.20 | 9.6 | ||
curated-fs (grep our notes) |
|||
| 0.75 | $1.74 | 4.3 |
The knowledge graph we were so proud of lost to grep over raw chat logs — while costing four times more and taking twice as many turns.
The failure analysis was more interesting than the score. We classified every wrong answer by grepping the store ourselves, and found zero curation loss: every fact needed by all twenty questions was present in the notes. The system wasn't forgetting. It was failing in three specific, fixable ways:
find
matched document titles only. Any fact not in a title cost the agent an expensive walk — retrieve a page, follow links, retrieve more — which is where the 9.6 turns and $5.20 went. Grep, meanwhile, just read the bodies.One failure taught us about representation, one about page architecture, one about tooling. The next three rounds fixed them in that order.
The curation prompt went through three revisions, each driven by a measured failure class. From v2 on, the tests ran on the second development conversation — one the prompt had never been tuned on.
v2 — date discipline. Record the relative wording with its resolution ("the Sunday before 25 May 2023 (21 May 2023)"), preserve the source's precision (a week stays a week; never invent day-precision the speaker didn't give), and cap entity pages at 300 words, moving detail onto focused sub-pages. Temporal J jumped from 0.60 to 0.92, and curation itself got 40% cheaper — focused pages are cheaper to maintain.
v3 — no pages about conversations. The curator had been creating "catch-up" pages dated by when a conversation happened, and those dates shadowed the dates of the facts discussed in them. New rule: an event is something that happened in the speakers' lives, dated by when it occurred — never by when it was mentioned. v3 also added relationship pages: for people who interact regularly, one page recording the threads they share — parallel experiences, joint plans, things in common. That rule came straight from a failure: "What do Jon and Gina have in common?" wants a synthesis (both lost their jobs, both started businesses) that lived scattered across two person pages.
v3 produced the first genuinely encouraging number: grep over our notes (0.85) now beat grep over the raw transcripts (0.75). The distillation was finally worth more than the source material — a property the published evaluations rarely show, since most memory systems score below their own full-context baselines. For a one-time curation cost of about $10, the notes had become a better substrate than the conversation itself.
One number refused to move, though. The curated
arm — IWE tools instead of grep, same notes — sat at 0.80 while grep hit 0.85. The relationship page held the answer to the common-thread question; grep found it; our own tools didn't. Identical store, 0.05 J apart, at nearly three times the cost. There was no longer anywhere to hide: the notes were good, and the read tooling was the bottleneck.
This is where the project started paying for itself. We had built the benchmark to understand how the tools actually perform as agent memory — and by now it was doing exactly that, with uncomfortable precision: an engineering instrument that priced each product gap in dollars and J. The failure analyses condensed into a ranked list — body-content search first, structured queries over MCP second, targeted editing third — and we started shipping down the list, re-measuring after each change.
BM25 full-text search. iwe_find
gained a real ranking engine: BM25 over title and body, maintained incrementally inside the document-ingestion pipeline, so every consumer — CLI, MCP server, editor — stays in sync as documents change. Re-running the bench: every failure we had classified as "fact buried in a body, unreachable by title search" flipped to CORRECT, on both development conversations. The multi-hop question that had haunted three rounds — the one whose answer sat on the relationship page — was finally reachable. The tools arm hit parity with grep at 0.85.
But parity at four times the cost. Turn counts fell (the agent found things faster), yet token spend rose — the culprit was our own output format, the link-rich result card find
printed for every hit. The bench had a new lesson: for agents, output rendering is a first-class cost, and the next gap was not what the tools could find but what they said.
The deepest tooling investment came next, and it started from a simple observation about the curator's economics: adding one fact to a page meant rewriting the whole document. That's how 9% of event pages in round one ended up structurally malformed — an agent re-emitting a full document under pressure will eventually misplace a frontmatter block — and it's why curation cost grew as the store did.
The fix became a language: block operations. IWE's query language could already select and shape documents; we extended it to address the structural nodes inside them — headers, paragraphs, list items, code blocks, tables — with one grammar used everywhere:
find --matches '(?i)todo'
greps every block in the store and answers with key › section path › text
lines. find --blocks PRED
returns matching blocks as structured data.$append
a line under a section, $replaceText
one line, $insertAfter
a block — each operator carrying its own selection predicate, validated fully before anything is written.Designing this for agents forced decisions that a human-oriented tool would never make. Four are worth recording:
Every mutation states its blast radius. Each operation can carry an expect
guard — "this edit should touch exactly one target" — and the operation fails loudly, listing what it actually selected, if reality disagrees. We then went a step further and split enforcement by surface: on the CLI, guards are opt-in (--strict
), because a human with git behind them shouldn't pay ceremony; over MCP — the agent surface — strict is always on, because the population most likely to skip the guard is exactly the one that needs it. The workflow this creates is locate, count, pin, mutate: find your targets, learn the count, write it into the edit, and the store refuses to run an edit whose shape surprises you.
Identity is a guard too. A document's key is a primary key: creating at an existing key is a conflict error, not a silent second copy. Let the agent set the key and derive it from metadata — the entity's name, the session's date — and a whole failure class disappears structurally: the same session cannot be recorded twice, a crashed ingest retries idempotently (the conflict error itself says "update instead"), and cross-links become computable — a timeline line can point at 2024-05-21
without a lookup, before that page even exists. We learned this from its absence: when keys were minted from title wording, every retry and rephrasing was a fresh identity, and the store quietly accumulated near-duplicates that no prompt admonition prevented. Like expect
, it converts a discipline ("search before you write") into an engine guarantee.
The guarantee is exactly as strong as the derivation is canonical: the first automated run created the same relationship page twice — joanna-and-nate
and nate-and-joanna
— until a canonicalization rule (names in alphabetical order) left the writer no choices. A uniqueness constraint dedups identities, not intentions.
Deleting a header is not deleting a section. Our first draft gave update operators subtree semantics — replace a header, get its whole section replaced. Hands-on testing showed this is a data-loss trap: an agent that "renames" a header by replacing it silently destroys everything beneath. The shipped semantics follow text-editing intuition instead: a header selection is the heading line; its contents re-attach and re-level, exactly as if you'd deleted the line in an editor. Destroying a whole section requires naming the section. The failure modes now point in the safe direction — the recoverable mistake (leftover visible content) instead of the silent one (a vanished subtree).
Error messages are the documentation agents actually read. A guard violation prints the count, every selected target, and the fix:
$replaceText expects 1 block, selected 2
jon › "He is wrapping up his business plan and actively searching for investors…"
jon-actively-searching-for-investors-27-may-2023 › "Jon is actively searching for…"
hint: narrow with $within or $matches, or raise expect
We watched agents recover from these errors in a single turn, using nothing but the message. When the strict layer names the two specific guards you're missing, a small model can drive a mutation language it has never seen.
With the block-operations tool (iwe_query
) exposed over MCP and the curator taught to use it — frontmatter set by a targeted $set
instead of a document rewrite, facts appended under sections instead of pages re-emitted — we re-ran everything on the second development conversation, ablating store and tools separately:
| store | answer tools | J | temporal | cost/20q |
|---|---|---|---|---|
| v3 store | read tools (previous best) | 0.85 | 0.92 | $4.30 |
| v3 store | + iwe_query |
|||
| 0.85 | 0.92 | $3.82 | ||
| new store | read tools | 0.85 | 1.00 | |
| $3.06 | ||||
| new store | + iwe_query |
|||
| 0.90 | ||||
| 1.00 | ||||
| $2.65 |
0.90 — eighteen of twenty, the best score any configuration had produced, with the temporal category perfect and cost down 38% from the previous best. The ablation told a clean story: the store improvements (the curator now keeps dated status lines — "As of 3 April 2023, Jon is expanding the studio's social media presence" — which resolved a temporal question that had survived every previous round) drove the accuracy; the tools drove the cost down; together they compounded.
Two quieter results mattered as much. The new store had zero structural defects — the malformed-frontmatter class that plagued round one is unrepresentable when metadata travels through $set
. And across forty answering sessions where the agent held a mutation-capable tool, the store diff showed zero writes: strict mode held in the field, not just in tests.
One prediction failed, and it failed instructively: curation cost didn't drop — the new store cost $10.98 to build against v3's $10.35. Targeted edits save money on large documents — but our own v2 rule had capped every page at 300 words, so there was nothing large to save on. The tool we'd built to fix a problem had arrived after a prompt rule had already worked around it. Which raised an uncomfortable question.
The 300-word page cap existed for exactly one reason: title-only search couldn't find facts inside big pages. That tool limitation was now gone — BM25 finds a fact in a page of any size, block projections return just the matching paragraph, targeted edits modify one line of a thousand-line document. The cap had become a fossil: a prompt rule compensating for a tool weakness that no longer existed, and causing the newest failure class — facts that belong together (why Gina started her store: passion and job loss) split across pages to satisfy a word count.
Auditing the curation prompt with that lens sorted every rule into three kinds. Tool workarounds — the cap, the fact-scattering rule — deserved deletion. Epistemic disciplines — date precision, occurrence-vs-mention dating, contradiction history — were the measured wins; they encode what memory means, not what tools can't do, and they took temporal J from 0.60 to 1.00. Drift guards — every page has one title, everything reachable from the index — stay because we measured agents violating them even when instructed.
The v5 prompt dropped the fossils and added their replacements: keep related facts together; a rich page is fine, the tools read and edit it block by block; record evolving state as dated status lines; never split one thought across pages.
Then we changed one more variable, the expensive one: the model. If the strict tool workflow really carries the structural burden — guards catching mistakes, operations validating before writing, errors teaching the fix — then the curator shouldn't need to be a frontier model. We re-curated the development conversation with Haiku, a smaller (~20 billion parameters) model, and measured everything again.
Haiku curated all nineteen sessions for $2.37 — 4.6× cheaper than Sonnet (~70 billion parameters), at about a minute per session — and produced a store with the same perfect structural hygiene: zero malformed pages, zero broken headings, full event/link machinery. A typical page from the $2 store:
---
type: event
date: 2023-01-28
participants:
- jon
---
Jon visited Paris. He described the experience enthusiastically and shared a photo from the trip.
The small model follows structural recipes faithfully — it never once wrote bad frontmatter, because the workflow makes that impossible. Stylistic advice is another matter, as the rest of this story will show.
Answering with Haiku over that store scored 0.80 — tied with Haiku-over-raw-transcripts on J, but with a tell in the category breakdown: perfect temporal, perfect single-hop, F1 twelve points higher (structured answers quote event titles nearly verbatim), and a collapse on multi-hop synthesis. Structure had compensated for the small model's retrieval and phrasing; nothing compensates for its reasoning.
So we split the roles. Haiku writes the memory; Sonnet reads it.
| curator | answerer | curation cost | J |
|---|---|---|---|
| Sonnet | Sonnet | $10.98 | 0.90 |
| Haiku | Haiku | $2.37 | |
| 0.80 | |||
| Haiku | Sonnet | $2.37 | |
| 0.90 |
The asymmetric pipeline tied the best configuration ever measured, at 22% of the curation cost. We had the conclusion drafted:
Structural curation quality is model-insensitive when the tools enforce it. Synthesis — at write time and at read time — follows the model. Spend accordingly.
Half of that paragraph survived the week. The next section is about which half.
Then the first validity rule earned its keep, at our expense. Reviewing the v5 prompt against the overfitting test — would this instruction make sense for any personal-conversation corpus? — we found that one of its worked examples, added to illustrate the dated-status-line rule, had been lifted from the development conversation itself. It contained the gold answer to one of the twenty benchmark questions. The curator was no longer question-blind; it had been handed 0.05 J in its own instructions.
So we did what the rules require. The store was quarantined in the run ledger, every prompt example was rewritten with synthetic entities and out-of-range dates, Haiku re-curated the conversation from scratch ($3.48 this time — the $2 curator became a $3.50 curator), and all three reading arms re-ran over the clean store.
| reader (same Haiku store) | contaminated run | clean rerun |
|---|---|---|
| Haiku | 0.80 | 0.80 |
| grep | 0.70 | 0.70 |
| Sonnet | 0.90 | |
| 0.75 |
Two results reproduced exactly. The one we had drafted conclusions about did not: the 0.90 fell to 0.75. The contaminated question was now honestly wrong, and two more flipped alongside it. The ledger's verdict, recorded next to the quarantined runs: the 0.90 was the high end of run variance plus one contaminated question.
Re-anchoring under the pinned test-configuration models completed the demotion: with claude-sonnet-4-6
answering and judging, the dev ladder came out raw-transcript grep 0.85, grep over the notes 0.80, the full IWE toolchain 0.70 — three arms a few questions apart, in the opposite order from the story we had been telling.
What survived every rerun: the structural finding — zero malformed pages, a property every store in this article has had since. What did not: the tie, and any belief that twenty questions can rank three systems sitting two or three questions apart. The first half of the drafted conclusion stood; the second half — where synthesis lives and where to spend — needed a bigger instrument.
The bigger instrument existed: the sealed test set. We froze the configuration — Haiku curation under the decontaminated v5 prompt, answering and judging pinned to claude-sonnet-4-6
— and spent the first installment: two conversations, 351 questions per arm.
| arm | J (351 q) |
|---|---|
| grep over raw transcripts | 0.812 |
| grep over the curated notes | 0.764 |
| the multi-turn tool agent | 0.735 |
Grep won on sealed data, and this margin was statistically real (z ≈ 2.4) — the frozen-config dev anchors had predicted exactly this ordering. The curated pipeline landed inside the published memory-system band (Mem0g 0.68, Letta's filesystem agent 0.74, full-context 0.73), but our own grep baseline sat far above the published baselines, so "beats published systems" would have been the wrong headline to take from it. The gap concentrated at single-hop (0.893 vs 0.802) — questions probing verbatim detail that curation had summarized away — and multi-hop, the category a knowledge graph should win, showed no curated advantage at all. The economics were just as one-sided: the tool agent read ~176k tokens over eight turns at $0.153 a question; grep read ~78k over four to five turns at $0.056.
We had built a system that was slower, costlier, and less accurate than grep, and this time there was no noise to blame. What the failure shapes argued was that the answer-time agent was the waste: if the notes are good, finding them should not take eight turns — the graph already encodes which pages belong together. So the reading architecture was rebuilt as a one-shot dossier: the harness runs a single retrieval — BM25 rank on the question, take the top pages, expand what they link to — inlines the result into the prompt, and the model answers in one turn holding zero tools. Everything graph-aware moves to write time and to that one retrieval call; nothing agentic remains at answer time.
To find the architecture's ceiling before pointing it at automated stores, we hand-built a reference store for the second test conversation: 34 pages, 7,500 words — one third the size of the raw transcript — written to the exact shape the curation prompt prescribes. (One disclosure: the hand-builder had seen some of that conversation's questions in earlier failure analyses, so read its number as a ceiling with an asterisk.) One-shot over the hand-built store scored 0.814–0.824 depending on the dossier mechanism — above raw grep's 0.809 on the same 199 questions, the first time any curated configuration had beaten the transcripts on test data — in one turn instead of about five, ~2 seconds median instead of ~12, at about a nickel a question. Temporal hit 0.95, the dated ledgers finally paying out.
Then we re-curated the same conversation with Haiku and pointed the same mechanism at the result: 0.72. The mechanism transfers; the store does not. Everything left between the automated pipeline and the ceiling was curation quality — and closing that gap is the rest of this story. From here on, the numbers are one 199-question conversation under the frozen judge: the ceiling is 0.814, raw grep is 0.809, and the automated store starts at 0.72.
The cheap curator has a ceiling, and we found it by trying to prompt our way through it: revision after revision left the automated store pinned near 0.70, against the hand-built 0.81, under an identical reading configuration. The failure analysis is not what we expected: the agent's episode pages are as good as the hand-written ones — sometimes better, with fuller verbatim quotes. The entire gap lives in the entity hub pages, and it survived three successive prompt versions, each of which fixed the measured defect and created the next. We capped hub size; the bloat migrated to the relationship page. We hardened link rules; compliance came back partial. We demanded terse timeline lines; the agent obliged by squeezing out the dates, and the temporal category — the store's crown jewel — collapsed.
The pattern underneath is worth stating plainly: the agent follows rule proxies; an editor optimizes the objective. Every prompt rule is a lossy compression of one goal — every fact findable by its words, or one link from a page that is, at a cost the reader can afford. When rules collide (terse versus dated, complete versus compact), the human resolves the collision by knowing why the rules exist. The agent picks whichever rule was stated most recently, and the defect moves somewhere else. Prompt iteration against this is whack-a-mole with a token budget.
So the next two moves attack the structure, not the wording. The first is a store lint: the store's conventions — key formats, per-page token budgets, line-length limits, dangling links, near-duplicate pages — declared once in the store's own config and checked mechanically, with violations riding back on every tool result as warnings the writer is expected to fix before closing the session. It is the expect
idea graduated from single mutations to whole stores: don't ask the agent to remember the conventions, make the store report its own violations, one write behind the writer. The second is a consolidation pass: the session-by-session curator is structurally append-only — it never revisits, so it can never merge a story arc that spans sessions, never prune a hub that grew wrong, never restore a date its own terseness dropped. One additional pass at the end of a conversation, rewriting the hubs with the whole store finally visible, gives the agent the one thing the human editor always had: hindsight.
Both experiments ran within the week of that diagnosis. The next sections are what they measured.
The lint shipped as store policy in config — token budgets per page kind, plus store-wide checks for dangling links, orphan pages, and near-duplicates — with violations riding back on every write and a full sweep before the curator closes each session. Calibration came first and taught its own lesson: the initial line-length caps flagged the hand-built store — exactly the content we want. Budgets that can't separate a known-good store from a known-bad one don't ship; page budgets separated cleanly, line caps didn't and were dropped.
The result was the largest single improvement of the automated era: one run with the lint loop did what three prompt versions could not. The hub page that no instruction could keep under control went from 2,408 words to 770 — the hand-built one is 767 — and the score jumped from 0.70 to 0.77 in one step. Temporal questions, which had collapsed when a terseness rule squeezed the dates out, recovered fully.
The curation logs revealed how guards win, and it isn't obedience. The curator fixed about half the violations and argued with the rest — "these warnings are expected here," "pre-existing conditions," "this is the intended architecture." Rationalization, in other words, from a model being paid $5.50 to file notes. But partial compliance was enough, because a linter differs from an instruction in one decisive way: it re-fires every session. Prompts decay as context grows; pressure that reapplies itself doesn't.
The consolidation pass — the hindsight experiment — failed twice and stays in the repo as a negative result: an editor pass rewrites wording, and search matches exact words, so every rewrite risks breaking a question→page match the original phrasing carried. Guards during writing beat editing after writing.
The guarded store also let us re-ask the question the retraction had left open: does synthesis really follow the reader? Haiku answering one-shot over its own store dropped only about 2.5 points — but spent a median of ~640 thinking tokens per answer where Sonnet spent 10, quadrupling latency and burning much of the price advantage in invisible reasoning. Writing memory tolerates a cheap model once guards keep it honest; answering from memory is where the strong model earns its cost.
The lint's page budgets had one blind spot: they see pages, not positions. A hub can sit legally under its total budget while its timeline section is 92% of the page — big-because-it-aggregates and big-because-it-re-tells look identical to a flat cap. So the store's conventions became document schemas: per-page-kind shape declarations — required sections by name, token budgets per section, bullets-only where prose means re-narration, a title pattern that keeps session pages dated — bound to key patterns and enforced as hard pre-commit gates on every write. A violating write is refused with the violation and the fix named, expect
-style; there is nothing to rationalize because nothing landed.
Schemas got the same calibration discipline as the lint, and it saved us again: a token cap per timeline entry flagged the hand-built store's best lines, so item budgets shipped loose, and the structural rule — bullets only, no prose blocks in a timeline — turned out to be the real re-narration detector. Thresholds need calibration; shape rules are robust on arrival.
The most striking schema result cost nothing to compute: validating three generations of stores against the same schema ordered them exactly as their QA scores had — the hand-built store at zero violations and 0.81, the lint-era store at 22 and 0.77, the pre-lint store at 272 and 0.70. The schema measures store quality without asking a single question, which is precisely what you want from a guard: the quality signal available at write time, not at benchmark time.
One defect survived every guard: hub timeline entries kept carrying the full title of the page they linked to — "(Screenplay completion and tournament friendships (23 January 2022))" where the curator had been told, in three successive prompt versions, to write "(23 January 2022)." We classified it as instruction-following failure, tightened the rule each time, and watched compliance stay partial and apparently random.
It was the product. IWE's renderer regenerates link text from the graph on every write that touches a page — so the curator wrote the date, and the tool expanded it to the target's title behind the curator's back, on every save, no matter what the prompt said. The "random" compliance had a perfect explanation: the only links the renderer couldn't rewrite were the ones pointing at pages that didn't exist yet. Obedience correlated with broken links.
The fix — preserve the link text the author chose instead of regenerating it — slimmed every hub mechanically and removed a copy of every episode's title vocabulary from exactly the pages that must not rank for episode questions. No curator of any intelligence could have closed that part of the gap; we had been blaming the model for the product.
There's a moral here about benchmarks. A unit test checks what you thought to assert; this defect was invisible to every test because the renderer was doing what it was designed to do. It took a benchmark exercising the full write path with an agent in the loop — and a hand-built store to diff against — to frame designed behavior as a defect. The benchmark that built the tools also debugged them.
With the renderer fixed and the guards in place, the automated pipeline's remaining gap collapsed into a single category. Single-hop questions: 0.874, exactly the hand-built store's number. Open-domain: exactly matched. Multi-hop: within one question. Temporal: 0.75 against the hand-built 0.95 — eight of the nine questions still separating the $4 curator from the human one, all in the category that asks when.
The per-question diff was surgical. Five of the eight were occurrence-date loss: asked when Joanna finished her screenplay, the automated store answered with the date the conversation mentioned it, because its timeline entries dated everything by session. The hand-built store had quietly used a pattern nobody had named: the event's date as the link text, the session's page as the target — "finished her first screenplay on 21 January 2022" — so the hub line itself answers when, and the link still points at the evidence. The other three were instance confusion: which tournament win was which, fixable by numbering the instances in the line ("won her second tournament…") so counting questions read straight off the ledger.
The date-as-link-text pattern has a satisfying property: it was impossible to produce until the renderer fix landed days earlier — the old write path would have overwritten the event date with a page title on the next save. The product change didn't just remove bloat; it made a new curation grammar expressible, and the prompt's job shrank to teaching it.
It closed half of them. The next run's curator adopted all three patterns from its first session, temporal recovered from 0.75 to 0.85, and the overall score reached 0.778 — the best automated number of the arc, 96% of the hand-built ceiling, from a curator that costs $4.45 a conversation. The progression tells the story of the whole guards era in one line: 0.68 with prompt rules alone, 0.70 after three rounds of prompt whack-a-mole, 0.77 the day the lint landed, 0.778 once the renderer defect was fixed in the product and the prompt could finally teach the pattern. What separates the automated store from the hand-built one is down to about seven questions of 199. Repeated answering runs over a fixed store differ by about two questions, so the gap is real — it is just, finally, small.
About memory. Capture is the easy part — every configuration we ever ran had the facts somewhere. What separates configurations is dating discipline (store uncertainty, not false precision), co-location (facts that answer questions together must live together), and dated status lines for evolving state. Temporal questions, the weakest category for every published memory system, went from 0.60 to 1.00 across the dev rounds on prompt rules alone.
About tools for agents. Output rendering is a cost model, not a formatting choice. Guards beat instructions — an expect
clause or a key-collision error catches what a prompt admonition never will, and the same holds at store scale: key formats, page budgets, and near-duplicates are checkable policy, not prompt hopes. Error messages are load-bearing documentation. And enforcement should live at the surface, not the grammar: humans get freedom, agents get strictness, the language means the same thing everywhere.
About enforcement layers. The guards sorted themselves into a taxonomy: schemas own shape (sections, budgets, block types — hard gates, nothing to argue with), graph hygiene owns links (dangling, orphans, near-duplicates — warnings that re-fire until fixed), and prompts own semantics (which date is the event's, what deserves a timeline line — the parts no pattern can check). Two laws attach. Every threshold must be calibrated against a known-good store before it ships — each one we tried flagged the good store first — while structural rules were robust on arrival. And agents negotiate with warnings but cannot negotiate with a refused write, so put the rules you cannot afford to lose behind gates and let the rest apply pressure.
About cost. One-time curation amortizes to nothing; per-question costs compound forever. The configuration of record — about $4.50 to curate a months-long relationship into a store, then one-shot answering at ~$0.05 a question — reflects that arithmetic. And raw-transcript grep, the baseline that never stopped embarrassing us, carries a hidden divergence: real conversation history grows without bound, while a curated store stays compact. (The strongest no-curation configuration in our records — IWE's own tools pointed at the raw transcript files — scores 0.80–0.85 across dev runs, and rides the same growth curve.) The benchmark's tiny corpora understate the curated arm's advantage, not overstate it.
About method. Until the comparison section below, every number here is a development number — twenty-question slices in the early rounds, one 199-question conversation in the late ones, single repetitions unless stated, an LLM judge with known artifacts (two of our persistent "failures" are answer-key defects we deliberately did not tune around). The test set's history is messier than the design, and we would rather record it than smooth it: the first installment was spent on the multi-turn configuration and told us grep wins; one of those spent conversations was then recycled as the late-arc development conversation; the second installment ran two conversations sealed until first contact, and their sealed-only score matches the overall number. Four test conversations remain unspent, reserved for the frozen configuration. We publish the harness, the prompts, the judge verbatim, the full run ledger — including the quarantined runs behind the retraction — and the per-question failure classifications precisely so the numbers can be distrusted productively.
The benchmark set out to test a thesis and kept disproving successive versions of it — once by catching a contaminated result we had already drafted the conclusions for — which is the only reason the final version is worth believing. Plain markdown, a graph of links, ranked search over normalized text, guarded block-level edits: it turns out that's not just a pleasant way for humans to keep notes. Measured honestly, it's a memory substrate a $4.50 agent can write under mechanical guards and a single retrieval call can read at 96 percent of a hand-built ceiling — one turn, two seconds, a nickel a question — and every page of it opens in your editor.
Two judge models appear in this article, both running Mem0's evaluation prompt verbatim, binary CORRECT/WRONG. The frozen test configuration answers and judges with a pinned claude-sonnet-4-6
, which re-scores the same calibration answers at 0.70 — the strictest grader on that shared anchor. Every number in this section is sonnet-4-6-judged. That choice makes our numbers comparable and conservative at the same time, and this section is where both properties earn their keep.
The published LOCOMO results — the reason this benchmark exists in the form it does — sit on approximately that scale:
| system | J | source |
|---|---|---|
| Mem0 | ~0.67 | published |
| Mem0g (graph memory) | 0.68 | published |
| full-context (no memory system) | 0.73 | published |
| IWE one-shot, automated ($4.50/conversation) | ||
| 0.75 | ||
| ours, test set, strict judge | ||
| grep agent over raw transcripts | 0.81 | ours — not a memory system |
Our row is 652 questions over four test conversations, two of them sealed until first contact; the sealed-only cut scores 0.754 against the 0.752 overall — the check that iterating on previously spent conversations did not inflate the number. Four more test conversations remain unspent. And since our judge is the strictest in the table on the shared calibration, the comparison above should, if anything, understate our rows.
What is comparable across setups is the economics, because tokens and milliseconds don't depend on the judge:
| IWE one-shot | grep agent | Our old multi-turn tool agent | |
|---|---|---|---|
| turns per question | 1.0 | 4.6 | 8.1 |
| context read per question | 20.8k tok | 77.8k tok | 176.5k tok |
| latency p50 / p95 | 1.9 / 6.5 s | 11.6 / 37 s | 21 / 59 s |
| answering cost per question | $0.051 | $0.056 | $0.153 |
| context growth as history grows | bounded by construction | linear | agent-dependent |
The bottom line we're comfortable defending: under the strictest judge in the table, a $4.50 question-blind curator writing plain markdown, read by one search-and-expand call, matches every published memory system on LOCOMO — and the strongest system on the benchmark is still a competent agent given grep and the raw transcripts, which says as much about the benchmark's scale as about memory systems. The numbers that should move a reader are in the economics table: one turn instead of five or eight, two seconds instead of twelve or twenty, a fraction of the tokens read — for answers that are just as accurate.
The harness, prompts, and per-run records live in the memory-bench repository.
IWE is open source, the documentation is available at [https://iwe.md]