{"slug": "show-hn-i-evaluated-file-vector-graph-and-rl-based-memory-frameworks", "title": "Show HN: I evaluated file, vector, graph and RL based memory frameworks", "summary": "A developer's evaluation of three agent memory frameworks—file-based, structured store, and reinforcement-learning-trained experience—found that the file-based approach, as implemented in OpenClaw's memory-core plugin, outperformed a custom hybrid structured store on the same agentic benchmark when both ran behind the same agent loop with the same local open-weight model. The structured store, combining vector, graph, and associative retrieval, was built by the author and is published with its spec, while the experience-based approach, exemplified by MemHarness, represents the state of the art but was not directly benchmarked in this post.", "body_md": "# The Shapes of Agent Memory – Files, Stores, and Experience\n\nAugust 12, 2026\n\nAn agent that remembers across sessions can keep its memory as curated markdown files, as an auto-mined structured store, or as trained experience. I measured all of them: files against a structured store under one fixed model, a store-only head-to-head across the structured lineages, and an experience bank on the agentic benchmarks where the state of the art trains memory into the weights.\n\nThree side-by-side memory shapes: a file-based index of markdown lines, a structured store of embedded units linked by a graph, and trajectories of agent experience with one successful episode ringed.\n\nAn agent that only remembers within one conversation is a stranger with excellent manners: it greets you warmly every single day, and it has no idea\nwho you are. The moment you want it to know your projects, your preferences, and the thing you told it last Tuesday, you need memory that outlives the\ncontext window. There are three common shapes a modern agent memory system takes ([Fig. 1](#figure-1)). Two are stores that sit beside a frozen model,\nand they anchor opposite ends of a design axis; the third moves the memory behavior into the model itself.\n\nThe first keeps memory as **files the model curates**: a short index plus topic files, written in plain markdown, read back by searching and reading\nthem like any other file. It is what a coding agent reaches for when it has a filesystem and no database, and it is what\n[Claude Code](https://code.claude.com/docs/en/memory), [Cline](https://docs.cline.bot/best-practices/memory-bank),\n[Cursor](https://docs.cursor.com/context/memories), and [Windsurf](https://docs.windsurf.com/windsurf/cascade/memories) ship today.\n[OpenClaw](https://docs.openclaw.ai/concepts/memory) is the most thoroughly worked-out version of it: its default `memory-core`\n\nplugin keeps a curated\n`MEMORY.md`\n\nbeside dated session logs, and it adds a background consolidation pass that the other file-based products do not have. The second keeps\nmemory as a **structured store**: every turn is mined into small atomic facts, embedded into a vector index, threaded into a temporal graph, and read\nback by ranked retrieval. It is what you build when memory is the product, and it is what the dedicated memory startups\n[mem0](https://mem0.ai/research), [Letta](https://www.letta.com/blog/agent-memory/), and [Zep](https://arxiv.org/abs/2501.13956) sell. The third keeps\nmemory as **experience the model is trained to use**: episodes still land in a bank, but everything that makes them memory, what to retrieve, whether\nto trust it, how to turn it into action, is trained into the acting policy by reinforcement learning. It is the agentic state of the art\n([MemHarness](https://github.com/KnowledgeXLab/MemHarness)), the shape the field reaches for when retrieval stops paying, and it is where this post\nends.\n\nIn this post, I measure which shape is better rather than argue it, which meant building the first two. The structured arm is a [hybrid](#hybrid) of\nthe [two structured lineages](#two-lineages-place-and-entity-and-time), plus a layer neither has: an associative graph learned from which places\nactually get retrieved together, so recall can reach an item the query never ranked. The [file-based arm](#file-based) is a reconstruction of a\nshipping coding agent’s auto-memory, traced claim by claim to public documentation and\n[published with its spec](https://github.com/a40-labs/memory/tree/main/systems/file-based), not a strawman written to lose. Both run behind the *same*\nagent loop, on the *same* local open-weight model, scored by the *same* judge on the same public benchmark, so only the memory layer can move the\nnumber, and the per-question rows, with the scripts that recompute each figure and an explicit ledger of the few published scores whose rows could not\nbe released, are in [a40-labs/memory](https://github.com/a40-labs/memory). Hosted models come in where fairness demands: the\n[head-to-head](#the-lineages-head-to-head) reads every store through one shared reader and judge, `gpt-4o-mini`\n\n, the same model the graph vendor’s own\nnumbers were scored with; the agentic experiment fields a frontier actor, `claude-sonnet-5`\n\n. The [trained shape](#experience-architecture) cannot join\nthe controlled comparison at all, because the training *is* the method: unplug its bank and you have a different policy, not a baseline. The\n[last section](#remembering-what-worked-the-agentic-benchmarks) meets it on its home ground instead.\n\n**TL;DR** The structured store beats files on accuracy and on token cost at once; files win where memory stays small, or where the right answer is “I\ndon’t know”. Against my own interest, the hybrid is statistically indistinguishable from a plain vector index on LoCoMo; paired on long-haystack\nLongMemEval-M the same two stores separate by 15 significant points in the hybrid’s favour, which is consistent with structure paying as histories\ngrow (the arms differ in more than structure, so the bundle is what is measured), and no single benchmark ranks memory systems. Swapping the model\nstack that reads and judges the memory moves the score further than swapping between any two of the stores that work, which is why numbers do not\ntravel between protocols. On the agentic benchmarks, retrieved experience paid only where the actor was weak with headroom left; where the task yields\nto reasoning, a frontier actor reaches the trained system’s bar with no memory at all, and where the reward has a shape only practice teaches, the\ntrained policy stands alone.\n\n## Store architectures\n\nBoth sit beside a frozen model and persist facts across sessions. The difference that matters is *who does the work, and when*.\n\n### File-based\n\nFile-based memory spends its budget at write time, through the model. After a turn, the model decides whether anything is worth keeping, and if so it\nedits a file: a new line in the index, or a paragraph in a topic file. The index is small on purpose, because it is loaded into context every session;\na common budget is the first 200 lines or so. Everything else lives in topic files that are *not* loaded until the model goes and reads them. Recall\nis therefore whatever the model can find by keeping the index in view and grepping the rest. There is no embedder and no ranker. The whole system is\nthe model’s own judgment plus a text search, which is exactly why it is so easy to ship: if you have file tools, you have this.\n\nThis approach is everywhere in shipping coding agents: Claude Code’s auto-memory (a per-project `MEMORY.md`\n\nindex over model-curated topic files), the\n[memory tool](https://platform.claude.com/docs/en/agents-and-tools/tool-use/memory-tool) primitive in Anthropic’s API, the community’s\n[Cline “Memory Bank”](https://docs.cline.bot/best-practices/memory-bank) and its descendants, and the automatic memories in Cursor and Windsurf. Keep\nit distinct from the *instruction*-file family (`AGENTS.md`\n\n, `CLAUDE.md`\n\n, `.cursorrules`\n\n), which is human-authored static context; the accumulated,\nmodel-written kind is what this post evaluates.\n\n### Structured\n\nStructured memory spends its budget at write time too, but not through the model. Every turn is mined into atomic units, each embedded and stored with dense and sparse vectors, and salient facts are threaded into a graph whose edges carry validity windows so a later fact can supersede an earlier one. A background pass consolidates duplicates and merges the graph. Nothing on the write path asks the model to reason; it is extraction and embedding. Recall is a ranked hybrid query, and the most salient units are preloaded before the first user word, so the agent often answers without searching at all.\n\nThis is what the dedicated memory startups sell: [mem0](https://mem0.ai/research) extracts facts into a vector-first store with an optional graph\nlayer, [Letta](https://www.letta.com/blog/agent-memory/) (formerly MemGPT) pages tiered memory in and out of context, and\n[Zep](https://arxiv.org/abs/2501.13956) builds a bi-temporal knowledge graph, the strongest form of the idea. All of them put an extractor and a\nranker where the file-based approach puts the model’s judgment and a grep.\n\nTwo design decisions fall straight out of this split, and they are the whole game:\n\n**What gets saved.** File-based memory saves what the model*chooses*to save. Structured memory saves*everything*, then ranks.**What gets found.** File-based memory finds what a literal search surfaces from an index that must stay small. Structured memory finds what a similarity ranker surfaces from a store that can grow without bound.\n\n#### Two lineages: place, and entity-and-time\n\nCalling all of that one “shape” hides a real split, because the structured pole has two lineages that organize memory around opposite primitives: **by\nplace, or by entity and time.**\n\nThe place-organized lineage files memory by *where it belongs* rather than by who it is about. [MemPalace](https://github.com/mempalace/mempalace) is\nthe cleanest public example: people and projects become wings, topics become rooms, and the original conversation text lives in drawers inside them,\nretrieved by semantic search scoped to a region rather than swept across a flat corpus. The defining choice is what it declines to do at write time.\nIt stores the text verbatim, and does not summarize, extract, or paraphrase, so ingest is an embedding and a filing decision with no model reasoning\nin it at all. Its structural weakness is aggregation: an answer scattered across many rooms depends on one ranked query surfacing all of it at once,\nand no artifact in the store has gathered it in advance.\n\nThe entity-and-time lineage (Zep’s [Graphiti](https://github.com/getzep/graphiti)) stores a knowledge graph instead: raw messages kept as ground\ntruth, LLM-extracted entity nodes that are resolved and deduplicated across sessions and carry maintained summaries, and one-line distilled fact edges\nbetween entities. Every edge holds validity timestamps, relative dates are resolved to absolute ones at ingest, and a contradicting new fact *closes\nthe old edge’s validity window* rather than deleting it. The reader receives distilled facts with date ranges plus entity summaries, never raw\nmessages. The cost inverts: an LLM reasons at ingest, on every message, for extraction, resolution, and invalidation.\n\nThree consequences follow ([Fig. 3](#figure-3)):\n\n**Aggregation.** An entity node accumulates every fact about a person*by construction*, so an enumeration question (“what are all of X’s hobbies?”) arrives with a pre-built aggregate. A place-organized store has no such artifact: it must hope one ranked query surfaces every scattered item, and the members of an enumerable answer are usually semantically far apart (running, pottery, and movie nights share little beyond the person), so no single query ranks them all into the top-k even when every item is in the store. Keep this weakness in mind for the results: the hardest questions for this study’s structured arm, on both benchmarks, are exactly the ones that assemble an answer from facts scattered across many sessions.**Time.** Supersession lives in the store for one lineage (validity windows the reader can trust) and in ranking heuristics for the other.**Cost.** Place-organized is cheap at write time and leans on retrieval; entity-and-time pays heavy LLM cost at ingest to make reading cheap and precise.\n\nNeither dominates. If your workload is scoped recall over evolving topics, the place lineage’s load layers are the better fit; if it is cross-session aggregation and “what is true now”, the graph lineage earns its ingest bill. Which raises the obvious question: why not take the cheap half of each?\n\n### Hybrid\n\nThe hybrid takes exactly that bargain, and it is the design this study measures ([Fig. 2](#figure-2)’s right panel is its architecture). Take the\nplace-organized store’s write path wholesale: atomic dated facts filed by location, embedded with no LLM reasoning, recalled by layered loads and\nranked hybrid search. Then borrow one thing from the entity-and-time lineage: validity windows, so a contradicting new fact can close an old one’s\nwindow instead of competing with it at recall time. That combination is not unique to this study, and it would be misleading to imply otherwise:\nMemPalace ships a temporal entity graph with validity windows of its own, alongside its rooms. What differs here is smaller and more specific, and it\nis the third layer below.\n\nWhat the hybrid buys is the cost profile of place with the time semantics of the graph: writes stay embedder-cheap, and “what is true now” questions\nget dated, supersedable facts rather than ranking heuristics alone. One temporal nuance the graph lineage does not spell out: supersession must\ndistinguish conflicting *states* from additive *events*. “Works at Acme” should close “works at Beta”; “scored 2 goals this week” must never close\n“scored 3 goals last week”, or counting questions become unanswerable. What the hybrid deliberately leaves out is the graph lineage’s expensive half:\nno LLM at ingest, so no entity resolution and no maintained summaries, and cross-session aggregation stays its structural weak point.\n\nThe design’s answer to that weak point is a third layer the two lineages do not have: an **associative graph learned from usage statistics**.\nLocations that co-occur in retrievals more often than chance predicts get linked (a statistical test, not embedding similarity: “these go together” is\na stronger claim than “these look alike”), and recall runs as *anchor, expand, fuse*: ranked search anchors on what it can find, the association graph\nexpands to linked locations the query never ranked, and the fused result caps the graph’s contribution ([Fig. 4](#figure-4)). The design principle\nunderneath answers both lineages’ disclosed failure modes at once: **the graph is only ever allowed to add candidates, never to displace the anchor\nset.** That guards the retrieval, not the reader, since added context can still distract the model downstream; what the measurements showed is no harm\nat the scale tested, not monotonicity. Where the entity-and-time store is hard-bounded by its extractor, the hybrid keeps raw dated facts as the\nanchor, so a graph failure degrades to plain ranked retrieval; and where pure place organization can never *reach* an item its one query failed to\nrank, expansion gives it a query-independent path there.\n\nOne scope note for honesty, and it is a large one: the benchmark protocol below writes facts directly into the store, which never triggers the background consolidation where supersession lives (the next subsection is about that pass), and the association graph starts empty. So the configuration actually measured is the hybrid’s place-organized core: dated facts plus hybrid ranked search. Read its scores as a floor, with one update from later work that cuts against my own design: when the association graph was subsequently seeded from real co-retrieval statistics and given one controlled, paired shot at exactly the aggregation failures it exists to fix, it changed nothing. No harm (the capped fusion held), but no recovery either. The associative layer stays a design capability, not a measured contributor.\n\n### Consolidation\n\nEvery architecture so far has been described by two paths, write and read, and every description is incomplete. There is a third path, and it is not a\nfourth architecture: a pass that runs between sessions and reorganizes what is already stored. The field calls it **dreaming**, after the\nconsolidation that happens in sleep, and it is the only path that can repair a store that is already wrong.\n\nIt cuts across the taxonomy rather than extending it. Files can be consolidated, place-organized stores can be consolidated, and the entity-and-time lineage consolidates so eagerly it is easy to miss: resolving a mention against existing entities and closing a superseded fact’s window is exactly this work, moved to ingest and paid per message. That relocation is the real choice on offer. Consolidate at ingest and every write pays for order the store may never need; consolidate in the background and writes stay cheap while the store carries its own mess until the pass comes around.\n\nThe two implementations below sit at opposite ends of that trade ([Fig. 5](#figure-5)).\n\n[OpenClaw](https://docs.openclaw.ai/concepts/dreaming) runs the file-based version nightly, in three phases borrowed from a night’s sleep: a light\nphase deduplicates the recent buffer, a phase named after REM (the rapid-eye-movement stage where human brains replay the day and connect it to older\nmemories) looks across conversations for recurring themes, and a deep phase promotes survivors into `MEMORY.md`\n\n, the index every session loads. What\nmakes it more than cleanup is the gate: an item earns its place by being *used*, clearing a score threshold and several recalls across distinct\nqueries. The store learns what matters from what the agent kept reaching for, a signal neither the write nor the read path can see.\n\nThis study’s hybrid does the same job on different material, in three tiers. Identical writes never duplicate, because a unit’s content hash is its\nprimary key. A minute after a conversation goes idle, a pass rebuilds that conversation’s index cards. Then once a day the whole store is swept: units\nare clustered by similarity, using the vectors they already carry, and each cluster collapses to its longest phrasing with the others’ provenance\nfolded in. The bar is set high so merely *related* facts stay apart, and because the pass reuses stored vectors instead of re-embedding, it spends no\nmodel calls at all. So OpenClaw promotes upward into a file the model reads; the hybrid collapses sideways into a store the ranker searches.\n\nTwo limits come with that design, and both are visible before any measurement. Cheap similarity catches restatements but misses the same event told in other words, and closing that gap means a model judging each pair, which puts per-item reasoning cost back into the one path that had none. And the value of tidying at all depends on the reader: a model answers correctly about a handful of restatements sitting in front of it, merged or not, so storage-level cleanup earns its keep only once duplicates outnumber what the reader can hold. Consolidation pays most on long histories read by weak models, least on short ones read by strong ones. The measurements here cover only the second regime: run at roughly fifty sessions per history it merged real duplicates and bought no accuracy, and the main experiment never triggers it at all, so consolidation contributed nothing to the structured arm’s scored numbers; whether running it would raise or lower them is unmeasured.\n\n## Experience architecture\n\nEvery store architecture above shares one assumption so basic it is easy to miss: **the model that uses the memory is frozen.** The store gets\nsmarter, better ranking, better structure, better time semantics; the reader of it does not. The agentic memory line of work drops exactly that\nassumption, and [MemHarness](https://github.com/KnowledgeXLab/MemHarness) ([paper](https://arxiv.org/abs/2607.28272)) is its cleanest current example:\nhold the store simple, and train the model’s *use* of it instead.\n\nThe bank half looks deliberately familiar ([Fig. 6](#figure-6)). After every episode the trajectory is summarized and written into a vector store with\nsemantic embeddings, deduplicated semantically, and periodically pruned by empirical utility, so entries that keep paying rent stay. By this post’s\ntaxonomy that is a structured store: inspectable, swappable, nothing a reader of the sections above has not seen.\n\nThe difference is everything downstream of retrieval. Where a store architecture hands retrieved items to a frozen model and hopes its judgment\nsuffices, the experience architecture makes that judgment the trained object. Acting is a five-stage policy: **observe** the current state;\n**retrieve** the top-k experiences, each paired with the source observation it was learned from; **critique** the retrieved experience against the\ncurrent state (does this actually apply here?); **reconstruct** it into state-specific guidance when it does, or reject it and fall back to\nself-reasoning when it does not; then **act**. The whole pipeline is trained end-to-end with reinforcement learning (GRPO, group-relative policy\noptimization over grouped rollouts, with format rewards that keep the retrieval and reconstruction stages from collapsing), cold-started from a couple\nhundred teacher-written memory records.\n\nWhy go to that expense? Because the untrained alternative is not merely weaker, it is negative: in their own ablation, handing the trained policy raw\nreplayed episodes instead of reconstructions makes it *worse* (76.4 with no memory to 70.1 with raw replay). Retrieval gets the experience into view;\nnothing about a frozen model guarantees the experience gets *used*, and a policy that has learned when to trust a memory and how to rewrite it for the\nsituation at hand is solving a problem that no amount of store engineering touches. The cost profile inverts accordingly: the store architectures pay\nat write or read time and bolt onto any model; the trained one pays in training compute and is inseparable from the one model it trained.\n\nThat is also why it cannot join the controlled comparison that follows: unplug its bank and you have a different trained policy, not a baseline. The honest meeting point is its home ground, the agentic benchmarks, where the final section takes the structured store to meet it.\n\n## Evaluation\n\nTwo public benchmarks carry the comparison: LongMemEval as the primary, LoCoMo as the second opinion. Each gets a subsection below, because what a benchmark measures, and what its numbers have been made to say in the wild, decides how much a score is worth.\n\nEvery result table below compares the same three arms, named the same way throughout. Each table and figure states the sample it was scored over,\nwritten out in the tables and abbreviated as **n** in the figures, meaning the number of questions, games, or sessions behind that number:\n\n**No-memory**: the same agent loop with the memory layer removed. The floor that sizes what memory contributes at all, and proof that the judge cannot be gamed by refusing everything.**File-based**: the markdown reconstruction described above. An LLM-curated index plus topic files, recalled by grep and read.** Structured**: the hybrid described above. Dated atomic facts, embedded on write with no LLM, recalled by ranked hybrid search.\n\nThe two pure structured lineages, place-organized and entity-and-time, do not run in the main experiment; they get their own store-only head-to-head at the end of this section, where the graph vendor’s production system competes through its own published retrieval.\n\nThe controls are the point. Both arms ran the same agent loop, the same locally served open-weight model as the answerer\n([ Qwen3.6-35B-A3B-mxfp4](https://huggingface.co/mlx-community/Qwen3.6-35B-A3B-mxfp4)), the same embedder where one was needed, and the same judging\npipeline: a model judge scores each answer against the gold one, and a deterministic pass then re-classifies refusals (saying “I don’t know” counts as\ncorrect only when the answer genuinely was not in the history). The judge is identical for both arms, which removes per-arm judge configuration as a\nfactor, though a shared judge can still prefer answer styles that correlate with an arm; it is also one model family scoring its own outputs. Both\nconcerns are flagged rather than hidden. The file-based arm is a faithful implementation of the documented approach (Claude Code’s auto-memory,\ndescribed above), with a couple of deviations that make it slightly\n\n*more*robust than the standard, not less. Anywhere the two arms could differ for a reason other than the memory architecture, I held them equal.\n\nOne thing this is *not*: a measurement of any shipping product. I reimplemented the file-based *architecture* and drove it with a local open-weight\nmodel, so these numbers say nothing about how Claude Code, Cline, or anyone else performs in their own product, on their own model. Naming products is\nabout where the architecture comes from, not a leaderboard of them. What is being compared is the memory architecture, with everything else held\nfixed.\n\nThat fixity is the whole reason to bother. Published memory numbers are notoriously hard to compare across vendors, and the LoCoMo subsection below tells the canonical story. A number is only worth anything when you know what was held constant. Here, everything but the memory architecture was.\n\n### LongMemEval\n\n[LongMemEval](https://arxiv.org/abs/2410.10813) is a public suite for long-term conversational memory: 500 questions spanning categories that separate\nthe easy from the hard: single-session recall, multi-session joins, knowledge updates (“what is the *current* value”), temporal reasoning, and\nabstention (knowing when the answer was never stated). Retrieval and answering both count: the system has to surface the right memory *and* answer\nfrom it, and an LLM judge scores the answer against the gold one. It comes in two sizes: **LongMemEval-S**, where each question sits over a history of\nroughly 47 prior sessions, and **LongMemEval-M**, the same questions over roughly 500-session haystacks. The main experiment runs -S; the -M numbers\nclose this subsection, because scale is exactly what they measure. It is the primary benchmark here because its haystacks are long enough to punish\nweak recall and its categories name the exact failure modes the architectures should differ on.\n\nEvery headline number below is measured on the **held-out** questions, never the tuning split; both arms scored fractionally *higher* on questions\nthey had never seen, which is the opposite of what overfitting looks like. How the 500 questions were split, what “tuning” concretely means, and every\nlimitation a skeptic should weigh (prompt heritage, reconstruction fidelity, self-judging, and an oracle control that splits the gap descriptively\ninto read-side and write-side halves) are collected in the [appendix](#appendix-methods-and-caveats). The one that matters most mid-read: the\nfile-based arm got an equal tuning budget, and two of its frozen fixes came from watching its own failures.\n\n#### Accuracy\n\n| Category (held-out questions) | Structured | File-based |\n|---|---|---|\n| Temporal-reasoning (91) | 0.802 | 0.407 |\n| Multi-session (97) | 0.608 | 0.330 |\n| Knowledge-update (36) | 0.833 | 0.528 |\n| Single-session-user (52) | 0.923 | 0.673 |\n| Single-session-assistant (44) | 0.568 | 0.273 |\n| Preference (18) | 0.611 | 0.333 |\n| Abstention (18) | 0.778 | 0.889 |\nOverall (category-reweighted) | 0.736 | 0.449 |\n\nOn the held-out questions ([Tab. 1](#table-1), [Fig. 7](#figure-7)), the structured arm (the hybrid) scored **73.6%** and file-based scored **44.9%**\n(category-reweighted; raw 73.1% and 44.1%). The paired difference is **28.7 points**, 95% confidence interval **[22.1, 35.4]**, comfortably clear of\nzero. Three checks say the result is solid rather than lucky: the number barely moved from the tuning set to the held-out set for either arm (both\nactually ticked *up*, the opposite of an overfitting signature); widening the held-out set from 256 to all 356 non-tuning questions changed the gap by\n0.0002; and throwing out every question where either arm’s answer was truncated by the serving layer still leaves 74.1% against 50.2%. For scale, a\nno-memory baseline answering the same questions with no memory at all scores **9.8%**, so both architectures are doing real work; the question is how\nmuch.\n\nThe category breakdown says *why*, and it is not subtle. The widest gap is **temporal reasoning** (80% against 41%), and **multi-session** (61%\nagainst 33%) is close behind and clearest about the mechanism: its answers are assembled from facts mentioned in several different conversations. A\nliteral search over a deliberately small index is the wrong tool for that. If the joining fact sits in a topic file the model never thought to grep,\nit is simply gone, and the model, to its credit, usually says it does not know rather than inventing an answer. Ranked retrieval over an unbounded\nstore does not have this failure mode: the fact was saved whether or not anyone predicted it would matter, and similarity, not a filename, brings it\nback. **Knowledge updates** (83% against 53%) tell the same story from another angle.\n\nThere is exactly one category the file-based arm **wins**: **abstention** (88.9% against 77.8%), knowing that something was never said. That is not a\nrounding artifact, and it replicates: on the second benchmark’s adversarial questions, in the LoCoMo section below, the file-based arm beats the\nstructured one by an even wider margin, and a no-memory baseline that refuses everything beats them both. The mechanism is the same in both places and\nit is worth stating plainly, because it cuts against the headline: **a store that remembers less over-answers less.** Ranked retrieval almost always\nsurfaces something plausible enough to tempt an answer, while a curation-limited store often has nothing to offer and the model correctly says so.\nEager retrieval needs an abstention discipline bolted on; sparse memory gets one for free.\n\n#### Cost\n\nAccuracy is half the story. The other half is what each answer costs, and here file-based memory pays twice: more tokens, for a worse answer.\n\nComparing cost honestly requires separating two currencies. Both arms spend **model tokens** (prompt plus completion through the 35-billion-parameter\nmodel, as the serving layer reports them; hidden reasoning is not always reported, so these are floors), and those are directly comparable. The\nstructured arm *additionally* spends **embedder tokens** (a 2-billion-parameter model producing vectors), which cost orders of magnitude less per\ntoken and have no counterpart on the other side. Summing them into one number would be meaningless, so I never do; nor does this ledger price latency,\nper-token rates, or infrastructure, so “cheaper” here means fewer measured model tokens, not a total cost of ownership.\n\n| Chat tokens per question | File-based | Structured |\n|---|---|---|\n| Writing memory (LLM curation, amortized per question) | 246.1k | 0 (verified) |\n| Answering (recall plus reasoning) | 40.4k | 19.3k |\nTotal | 286.5k | 19.3k |\nTotal per correct answer | 665k | 27k |\n| Embedder tokens per question (estimated; separate currency) | 0 | 107.8k |\n| Wall-clock per ~50-session ingest | ~35 min | ~5 min |\n\nIn model tokens ([Tab. 2](#table-2), [Fig. 8](#figure-8)), per question: file-based **287k** against structured **19k**. Divide by accuracy to get the\ncost of a *correct* answer, which is what you actually pay for, and it is **665k against 27k**. On top of its 19k, the structured arm spends about\n**108k embedder tokens** per question on the write path. Even charging those at par with model tokens, which wildly overstates them, it remains the\ncheaper architecture. The reason is the write path ([Fig. 9](#figure-9)).\n\nCurating a file is a *reasoning* act. For every session in a history, the model reads the current index, decides what is worth keeping, and rewrites a\nline. Over a full ingest that came to roughly **246k model tokens per history** and about **35 minutes** of wall-clock per history on my hardware.\nStructured memory writes by embedding, no model in the loop, which finished the same history in about **5 minutes**, roughly a sevenfold speedup on\nthe write path. The two write costs are in different currencies (one is LLM chat tokens, the other is embedding-model tokens), so I never subtract one\nfrom the other, but the direction is not close.\n\nThe read paths differ too, in a way that compounds. File-based recall is iterative: keep the index in view, grep, read a file, maybe grep again, then answer. That longer, multi-round path also turned out to be more fragile. Under a busy serving layer the file-based arm hit truncation on 20 of 144 answers against the structured arm’s 3, precisely because it asks the model to generate more, over more rounds, with more chances to be cut off. Some of that is my serving setup, but part of it is intrinsic: a longer read path has more surface to fail on.\n\n#### The long haystack: LongMemEval-M\n\nThe -M variant asks the same questions over roughly ten times the history. The two store-only rows in [Tab. 3](#table-3) ran on the same 100-question\nsample, drawn once by seed before either arm ran (the drawn ids are published in the repo; the ordering of draw and runs rests on the study log), one\nretrieval and one reader call each under the benchmark’s official per-category judging, so their comparison is paired even though neither is paired\nwith the main experiment:\n\n| System (LongMemEval-M) | Score | Questions scored |\n|---|---|---|\nHybrid (store-only) | 0.750 | 100 (pre-drawn sample) |\n| Place-organized (MemPalace, store-only) | 0.600 | The same 100 |\n| Hybrid (full agent loop) | 0.632 | 500 (complete set, different harness) |\n| File-based | None | None: another ~2-3 days of runs at 10x the history |\n| Entity-and-time (Graphiti OSS) | None | None: ~600 single-stream GPU-days, or ~$7,000, to ingest |\n\nLong histories are the regime structure exists for, and the paired rows put a number on the claim: fifteen points, rescuing 22 questions against losing 7, exact p = 0.008 under the same paired test as the head-to-head below. The gap clears the sample’s own confidence interval, and the shape of the win matches the mechanism, with the hybrid sweeping the single-session categories (14/14 and 11/11) and pulling ahead on the multi-session and temporal content that long haystacks exist to test.\n\nStore-only gave the reader one ranked context and outscored the full loop on -M (0.750 vs. 0.632) and the 100-question -S sample in [Tab. 7](#table-7)\n(0.80 vs. 0.72). Prompt and loop effects were not isolated, so both gaps are directional.\n\nGraphiti OSS is unscored because ingesting the benchmark would be prohibitively expensive: it requires a reasoning call for each of the haystack’s 3.7 million messages, while an embedder takes milliseconds. At the roughly 14 seconds per message measured on this hardware, that is about 600 days of single-stream GPU time; parallel serving divides the wall-clock but not the bill, and renting a small hosted model to do the same work would have cost roughly $7,000 at list prices. I was not willing to spend either on one row of one table, so the row stays empty and the reason is published.\n\nThe file-based row is empty for a simpler reason: runtime. Fifty questions over ten times the history is another two to three days of runs, which fell outside this study’s window. Those runs were the only test of what this post’s main comparison implies about scale — that files fall further behind as histories grow — so that claim is measured at roughly 47 sessions and untested at 500, precisely where I expected the gap to be widest. Untested is not refuted, and the long-haystack rows above are a different pair: they say nothing about how files would have done. My guess is that the gap widens rather than narrows, and mostly on the write side, since an index capped at a couple of hundred lines cannot grow tenfold with the history behind it — curation drops more of what was never written down, while a ranked store simply retrieves from a larger pool. A guess is all that is, though, and the run that would settle it remains outstanding.\n\nThat is not a knock on the lineage so much as a statement of what it costs to reach the regime that matters. -M is where real assistants drift, and the paired rows above show it is where the benchmarks disagree: the flat store that is indistinguishable from the hybrid on LoCoMo falls 15 points behind here, where multi-session organization starts to pay.\n\n### LoCoMo\n\nA single benchmark is a single opinion, so the same three arms (no memory, file-based, structured) also ran on\n[LoCoMo](https://arxiv.org/abs/2402.17753), the other widely used long-term-conversation suite: 10 very long two-speaker conversations, each spanning\ndozens of sessions, with 1,986 questions across single-hop, multi-hop, temporal, open-domain, and adversarial (unanswerable) categories.\n\nLoCoMo needs its story told before its numbers can be trusted, because it is the benchmark on which the field’s most public scoring fight happened.\nZep reported 84% on it. mem0’s CTO [filed an issue against their evaluation code](https://github.com/getzep/zep-papers/issues/5) arguing the real\nnumber was 58.44: the adversarial category had been counted in the numerator but excluded from the denominator, and the baseline configurations\ndiffered. Zep’s [rebuttal](https://blog.getzep.com/lies-damn-lies-statistics-is-mem0-really-sota-in-agent-memory/) re-ran with the error fixed and\nreported 75.14, while pointing back at mem0’s own reporting (whose LoCoMo figure has been cited\n[at both 67% and 92.5%](https://arxiv.org/abs/2504.19413) depending on the write-up). One system, one benchmark, three published numbers spanning 25\npoints, and the memory architecture never changed: the swing came entirely from scoring conventions, judge choice, and which categories count.\n\nSo why keep the benchmark? Because the dispute indicts the reporting, not the questions; because it is the suite the vendors actually compete on, so\nresults on it travel; and because the fight teaches exactly this study’s premise, that a number means something only inside a fixed, published\nprotocol. The dispute constrains the protocol here in three ways. Every score is published under both scopes, with and without adversarial, because\nwhether to count that category is precisely the axis Zep and mem0 fought over. The analysis resamples whole conversations rather than questions,\nbecause LoCoMo’s questions cluster inside just 10 conversations and pretending otherwise makes intervals too tight. And I evaluate a disclosed,\nseeded, category-stratified sample of roughly 30 questions per conversation. The rest of the benchmark’s fine print (retrieval recall confused with\nanswer accuracy in the wild, saturation critiques) lives in the [appendix](#appendix-methods-and-caveats).\n\n#### Accuracy\n\nThe run completed ([Tab. 4](#table-4), [Fig. 10](#figure-10)), and the result is the study’s most honest one, because the verdict depends on the scope\nin exactly the way the dispute predicts:\n\n| LoCoMo (300 questions, cluster CI) | No memory | File-based | Structured |\n|---|---|---|---|\n| All questions | 0.217 | 0.387 | 0.497 |\n| Excluding adversarial | 0.017 | 0.356 | 0.561 |\n\nExcluding adversarial (the scope the benchmark’s own convention arguably prescribes), the structured arm wins clearly: the paired difference is +0.205\nwith a cluster confidence interval of [+0.063, +0.356], and it dominates the memory categories (temporal 0.694 against 0.306, open-domain 0.656\nagainst 0.410). Include adversarial and the verdict collapses to a statistical tie (+0.110, CI [-0.007, +0.240]), because the file-based arm abstains\nbetter on unanswerable questions (0.508 against 0.246): its curation-limited store simply has less material to over-answer with, while ranked\nretrieval almost always surfaces *something* plausible enough to tempt an answer. Even the no-memory baseline “wins” adversarial outright (1.000) by\nrefusing everything, which is why a blanket-refusal system still only scores 0.217 overall. The lesson generalizes: eager retrieval needs an\nabstention discipline, and a store that remembers less over-answers less. Both readings are published; neither is smoothed away. One disclosure: the\nfile-based arm’s run predates a serving-layer retry fix, and 18 of its 300 answers died to output truncation and count as wrong under the symmetric\nrule; its numbers are floors.\n\n#### Cost\n\nThe cost asymmetry survives the second benchmark, at a smaller scale ([Tab. 5](#table-5), [Fig. 11](#figure-11)). LoCoMo’s conversations are far\nshorter than the primary benchmark’s haystacks, so the file arm’s curation bill shrinks, but the ordering does not change: 22k model tokens per\nquestion against the structured arm’s 12k, and 58k against 23k per correct answer, with each conversation’s ingest amortized over its sampled\nquestions. The structured write path again spends zero LLM tokens (verified against the serving ledger) plus about 0.8k embedder tokens per question\nin its separate currency. The gap compressing from roughly fifteenfold to roughly twofold is itself the finding: write-time curation is priced by\nhistory length, which is the primary benchmark’s cost story wearing smaller numbers.\n\n| Chat tokens per question | File-based | Structured |\n|---|---|---|\n| Writing memory (amortized per question) | 4.9k | 0 (verified) |\n| Answering (recall plus reasoning) | 17.6k | 11.6k |\nTotal | 22.4k | 11.6k |\nTotal per correct answer | 58k | 23k |\n| Embedder tokens per question (separate currency) | 0 | 0.8k |\n\n### The lineages, head-to-head\n\nEverything above compares files against one structured design, and it leaves the lineage question open: inside the structured shape, does the graph\nearn its ingest bill? The head-to-head answers it in the most controlled frame available: strip every system down to its retrieval and hold everything\nelse constant. Each store contributes exactly its top-20 results for the same 1,540 non-adversarial LoCoMo questions; one fixed reader (gpt-4o-mini)\nanswers from that context alone, one fixed judge scores it, and every store’s row is produced by the same script. The entity-and-time lineage appears\ntwice: as [Zep’s published retrieval contexts](https://github.com/getzep/zep-papers), their production system’s real output (and a fairness note they\nare owed: their published 75.14 reproduces from their own artifacts; 0.7461 is the same context re-scored under this unified frame), and as their\nopen-source engine [Graphiti](https://github.com/getzep/graphiti) run end-to-end on their paper’s recipe, both embedders it names. These rows sit far\nabove the bare-loop table above because everything about the frame differs; they are comparable to each other and to nothing else. For orientation,\n[Tab. 6](#table-6) puts the four methods side by side (pure place-organized differs from the hybrid only by dropping the temporal layer, so the hybrid\nbounds it closely):\n\n| Method | Write path | Read path | Time handling | Aggregation | Where measured |\n|---|---|---|---|---|---|\n| File-based | LLM curates markdown | Index in context + grep/read | None built in | Index + luck | Main experiment |\n| Place-organized | Embed and file, no LLM | Layered loads + ranked search | Ranking heuristics | Ranked-query hope | This head-to-head |\n| Entity-and-time (Graphiti) | LLM extracts, resolves, invalidates | Distilled facts + entity summaries | Validity windows in the store | Entity nodes, by construction | This head-to-head |\n| Hybrid (place + time) | Embed and file, no LLM | Ranked search; associative expansion in the design | Dated facts; windows in the design | Ranked-query as measured | Main experiment and here |\n\n#### Accuracy\n\n| Store (one reader and one judge throughout) | LoCoMo (1,540 questions) | LongMemEval-S (100 questions) |\n|---|---|---|\n| Hybrid (this study’s structured arm) | 0.7825 | 0.80 † |\n| Place-organized (MemPalace, dense over raw turns) | 0.7792 | 0.60 † |\n| Entity-and-time (Zep Cloud, published contexts) | 0.7461 | None |\n| Entity-and-time (Graphiti OSS, two embedder configs) | 0.5338 / 0.5286 | 0.35 |\n\nFour findings ([Tab. 7](#table-7), [Fig. 12](#figure-12)).\n\n**The hybrid beats the graph vendor’s published retrieval**, and the difference is real under a paired test ([McNemar’s test](https://en.wikipedia.org/wiki/McNemar%27s_test), which scores only the questions the two systems disagree on; p < 0.01). So, separately, does the plain dense store: raw turns beat distilled facts here before any hybrid machinery is added at all.**The hybrid is statistically indistinguishable from the flat vector index**(+0.3 points, paired CI95 [-1.8, +2.4]), which is the uncomfortable finding and belongs in the open: no detectable gain from place-plus-time on this benchmark, though the interval allows small effects either way. It is also only half the story: the same two stores, paired on LongMemEval-M’s long haystacks ([Tab. 3](#table-3)), separate by 15 points (0.750 against 0.600, p = 0.008). The two arms differ in more than structure (fusion, reranking, recency, atomic facts against raw turns), so read that as this implementation beating that one at long histories while being indistinguishable from it at short ones: the benchmark-disagreement point again, made by one pair of systems.**The distilled pipelines lose to the raw-text pipelines.** Both graph rows hand the reader LLM-distilled facts and entity summaries; both trail every raw-turn store, and the strongest graph row loses 3.3 points to the flat index while spending six times its context. The pre-built entity aggregates that make the lineage attractive for enumeration questions do not surface as a net win anywhere in this table; whatever they recover, the distillation loses more elsewhere. This is the single-session regression Zep itself discloses, visible benchmark-wide once the reader is held constant.**An LLM-at-ingest design is bounded by its extractor, and the extractor bill is real.** Graphiti OSS with the vendor’s own models lands around 0.53, so much of its 21-point gap to the vendor’s cloud contexts sits outside the extraction model’s capability. Driven instead by this study’s local 35-billion-parameter model, the same engine collapses to 0.29 (a study-log run whose per-question rows are not among the published data), and the mechanism is visible at ingest: it extracts several times fewer facts per message than the cloud output implies. A store that only knows what its extractor wrote down starves quietly.\n\nThe LongMemEval-S column of [Tab. 7](#table-7) is the same three stores under the benchmark’s official per-category rubric on a pre-drawn 100-question\nsample (the hybrid’s 0.80 there is a single retrieval and a single read under a shared reader prompt, which is why it sits above the same system’s\nagent-loop 0.72 on this sample; [Tab. 3](#table-3)’s discussion unpacks that ordering). Read the two columns across and the point makes itself: the\nflat store that ties the hybrid on LoCoMo trails it by 20 points on LongMemEval, because LoCoMo mostly rewards verbatim lookup inside a few dozen\nsessions while LongMemEval forces multi-session organization. The two benchmarks disagree about the same pair of systems, in opposite directions. No\nsingle benchmark ranks memory systems.\n\nThe same frame also measures the thing this post keeps insisting on, and it is worth putting a number on rather than gesturing at. Evaluating\nbyte-identical retrieval with the local 35B as both reader and judge instead of gpt-4o-mini doing both moved the hybrid’s LoCoMo score by 6.9 points\n(0.7130 against 0.7825); both roles change together, so the swing belongs to the evaluation stack as a whole, not to reader quality alone. Separately,\nre-judging identical answers under different judge prompts moved category-level accuracy by 5 to 15 points. Set that against the architecture\n([Tab. 8](#table-8)): the hybrid and the flat index differ by 0.3 points, and both sit 3.3 to 3.6 from the hosted graph. **On this benchmark, changing\nthe stack that evaluates the memory matters more than changing which of those three stores you built.** Only Graphiti OSS sits further away than the\nstack does, and by a lot: even the cheapest route to it, Zep Cloud to Graphiti OSS, costs 21.2 points, three times the stack’s swing. The scope\nmatters, because the claim inverts with the haystack: on LongMemEval-M the same hybrid-versus-flat pair that differs by 0.3 here differs by 15\n([Tab. 3](#table-3)), better than twice that swing. The evaluation stack dominates where the stores tie; the store bundle dominates where the\nbenchmark actually stresses it.\n\nThat last gap needs care, because three different things sit behind that number and conflating them makes it unreadable:\n\n**The reader** sits at the end and answers from whatever context it is handed. It is identical for every store, which is what makes the store comparison valid at all; the 6.9-point row swaps it together with the judge.**The extractor** sits at the start, inside the store, and decides what ever gets written down. It is part of the store being compared, not part of the harness around it.**The pipeline** is what the extractor runs in: how many passes it makes over each message, what it resolves, what it keeps.\n\nSo which of the three explains the 21 points between the last two rows of [Tab. 7](#table-7), the hosted Zep Cloud at 0.7461 and the self-hosted\nGraphiti OSS at 0.5338? Not the reader: it is the same model for both. Not the extraction model’s capability either, because the open engine ran with\nthe same class of extractor the vendor’s published numbers were built with and still landed where it did. What is left is everything the hosted\nservice does around that extractor, and this is the point where the comparison reaches the edge of what it can honestly claim.\n\nZep Cloud is a hosted product, and its row here is Zep’s own published context. I can measure what reaches the reader on each side, not the ingestion\nor ranking that produced it. The open engine’s contexts are visibly thinner, carrying fewer stored facts and much shorter entity summaries, but that\nis a property of the released artifacts rather than a description of anyone’s internals. One known difference does not favour them: their published\ningestion reads a `blip_captions`\n\nkey where the LoCoMo field is `blip_caption`\n\n([zep-papers#9](https://github.com/getzep/zep-papers/issues/9)),\ndropping image captions that my runs kept.\n\nNone of this suggests their published number is wrong. It reproduces from their own released grades, and their contexts re-scored under this study’s\nreader and judge give the 0.7461 in [Tab. 7](#table-7), within a point of their published 75.14. The 21-point gap is real and reproducible; its cause\nis not observable from outside. Read “the hosted pipeline” as a label for the part I could not see, not a mechanism I verified. A store can only\nanswer from what its extractor wrote down, however good the reader in front of it, and that is why importing a number from someone else’s protocol\ntells you nothing.\n\n| What changed | From | To | Points lost |\n|---|---|---|---|\nThe reader+judge stack, retrieval byte-identical | `gpt-4o-mini` 0.7825 | `Qwen3.6-35B-A3B-mxfp4` 0.7130 | 6.9 |\n| The store, reader unchanged | Hybrid 0.7825 | Place-organized 0.7792 | 0.3 |\n| The store, reader unchanged | Hybrid 0.7825 | Zep Cloud 0.7461 | 3.6 |\n| The store, reader unchanged | Place-organized 0.7792 | Zep Cloud 0.7461 | 3.3 |\n| The embedder inside one store | Graphiti OSS 0.5338 | Graphiti bge-m3 0.5286 | 0.5 |\nThe store, the cheapest route into the open engine | Zep Cloud 0.7461 | Graphiti OSS 0.5338 | 21.2 |\n\n#### Cost\n\nThe graph lineage pays twice ([Tab. 9](#table-9), [Fig. 13](#figure-13)). At read time, distillation was supposed to buy density, but the graph rows\nhand the reader six times the context of the raw-turn stores (21.5k chars median against 4.0k and 3.5k) and scores lower with it. Graphiti OSS is\nleaner at 7.9k and scores lower still, so density alone is not what the hosted pipeline is buying. At write time, the raw-turn stores embed while the\ngraphs run an LLM over every message, and that difference compounds brutally with history length. Each message costs the graph a reasoning call, or\nseveral: extract the entities and facts, resolve them against the entities already in the store, then check whether the new fact invalidates an old\none. Measured on this hardware that ran about 14 seconds per message, against a few milliseconds to embed one. LongMemEval-M’s haystacks hold roughly\n3.7 million messages, so the arithmetic lands at about 600 days of single-stream GPU time to ingest one run (parallelism divides the wall-clock, not\nthe bill), against hours for the embedding-only stores, which is why the graph lineage has no long-haystack row at all.\n\nSix hundred single-stream days is a fact about my hardware, not about physics, and it is worth saying so plainly because the hosted vendors do not wait for it: the work parallelizes almost perfectly, so a hundred concurrent workers bring the wall-clock to about six days. What no parallelism touches is the unit economics. Every message costs the graph several LLM calls where the raw-turn stores cost one embedding, and at list prices for a small hosted model that is roughly two orders of magnitude more per message: about $14 of ingest for a single long user history, against about $0.03. Graph memory is affordable; it is just priced like a product decision rather than an implementation detail, and the bill scales with everything your users ever said. Context sizes are medians over the same 1,540 questions; the ingest comparison is wall-clock on identical hardware.\n\n| Store | Median context / question | Ingest |\n|---|---|---|\n| Hybrid | 4.0k chars | Embedder only: ~$0.03 per long user history |\n| Place-organized (MemPalace) | 3.5k chars | Embedder only: ~$0.03 per long user history |\n| Entity-and-time (Zep Cloud) | 21.5k chars | LLM per message: ~$14 per long user history |\n| Entity-and-time (Graphiti OSS) | 7.9k chars | LLM per message: ~600 single-stream GPU-days, or ~$7,000, per -M run |\n\n## What file-based memory is actually good at\n\nA fair comparison has to state the other side, because file-based memory is popular for real reasons, and none of them are refuted by the numbers above.\n\n**It is human-readable and human-editable.** Your memory is a folder of markdown files. You can open it, read it, fix a wrong fact, delete a stale one, or commit it to git. A vector store is opaque by comparison. For a tool you operate yourself, this is worth a great deal.**It has zero infrastructure.** No embedder, no vector index, no background workers, no graph. If your agent already has file tools, memory is free to add. Structured memory is a small distributed system you have to run.**Its writes are distillation.** Because the model decides what to keep, each memory is a considered lesson, not a raw fragment. At small scale that curation produces a genuinely tidy, high-signal store, which is exactly the regime a personal coding assistant lives in.**Its reads are cheap when the store is small.** Everything above is the large-history regime the benchmark stresses. When the whole memory fits in the index, the read path is just the index in context, and the grep never fires. A few hundred lines of curated notes covers a lot of everyday use.\n\nRead the accuracy numbers with that scope in mind. The benchmark deliberately lives in the hard regime: dozens of sessions, facts scattered across\nthem, questions that force a join. That is the regime where curation forgets and a small index cannot hold enough, and it is the regime a memory\n*product* has to survive. It is not the regime a single-project assistant with fifty lines of notes lives in, and there it is not just adequate, it is\nthe better engineering trade. The regime-dependence cuts both ways, and it is worth being honest about: on easier, factual-recall benchmarks the gap\nnarrows sharply. Letta, a memory startup, [reported 74% on LoCoMo](https://www.letta.com/blog/benchmarking-ai-agent-memory/) using nothing fancier\nthan plain files, and argued a filesystem may be most of what you need. The structured store earns its keep specifically where the questions force\njoins across many sessions, which is the part of the problem I find most interesting and the part a memory product cannot dodge.\n\n## Remembering what worked: the agentic benchmarks\n\nEverything above measures one kind of remembering: what was said. An agent accumulates the other kind too, what *worked*: the know-how of past\nattempts. The natural question is whether this post’s architectures carry over, so the same structured store was put to work on the agentic benchmarks\nthe memory-training literature uses: [ALFWorld](https://alfworld.github.io/) (household tasks in a text world: find the mug, heat it, put it away) and\n[WebShop](https://webshop-pnlp.github.io/) (find and buy the right product in a catalog, scored with partial credit). Memory here is an **experience\nbank**: training-split episodes distilled into atomic entries (a task pattern, the moves that worked), embedded, and retrieved top-k into the acting\nmodel’s prompt. Nothing is trained; it is the same architecture as the conversational study, wearing different content.\n\nThe bar in this realm is MemHarness, the experience architecture described earlier: the 7-billion-parameter policy whose retrieval, critique, and reconstruction were trained by reinforcement learning. Being trained rather than bolted on turns out to be the whole story. Two untrained actors ran with and without the experience bank, everything else frozen, and each benchmark gets its own subsection below, mirroring the evaluation section. Two reading notes apply to every table: MemHarness’s out-of-distribution (OOD) number is the comparable one, since the untrained rows run unseen splits; and “no memory” removes only the experience bank, the harness’s task scaffolding (playbooks, target hints, loop guards) stays in every untrained row, so the ablation isolates retrieved memory. There is no cost subsection here, deliberately: the three pay in currencies that do not share an axis (self-hosted GPU time, API dollars, training compute), and charting the two we measured would imply a comparison the third cannot join.\n\n### ALFWorld\n\n[ALFWorld](https://alfworld.github.io/) is a text world of household tasks: find the mug, heat it, put it away. Six task categories, binary success\nper game, scored as macro SR (success rate averaged over categories, so no category dominates). The evaluation runs the 134 unseen games.\n\n#### Accuracy\n\n| Actor | Macro SR |\n|---|---|\n| Local 35B, no memory | 0.603 |\n| Local 35B + experience bank | 0.645 |\nFrontier actor (`claude-sonnet-5` ), no memory | 0.959 |\n| Frontier actor + experience bank | 0.973 |\n| MemHarness (GRPO-trained 7B) | 0.852 / 0.859 OOD |\n\nThe ablation runs at both actor tiers, and the deltas line up as the headroom pattern predicts: the weak actor gains 4.2 points from the experience bank, the frontier actor 1.4. Only the direction is claimable, though. The weak actor’s gain rescued 16 games and cost 10, which an exact paired test puts at p = 0.164, inconclusive under the pre-set threshold, so ALFWorld corroborates the pattern without carrying it; the significant weak-actor evidence is WebShop’s alone. One detail inside that null is worth keeping: five of six categories move positive with memory, while the category the 35B fails by looping is unmoved to three decimals. Retrieved experience does not repair a policy-level failure mode.\n\nAt frontier quality the benchmark saturates instead ([Tab. 10](#table-10), [Fig. 14](#figure-14)): the tasks yield entirely to strong reasoning\n(ten-step solves, four of six categories perfect), memory rescued exactly 2 games and hurt none (p = 0.5), and the untrained frontier baseline exceeds\nMemHarness’s number. That last fact is not claimed as a beat: it is actor class plus harness scaffolding, not a method comparison. The right reading\nis that ALFWorld’s bar is procedural competence, find the object, use the appliance, with a ceiling any sufficiently strong actor reaches, and there\nare two routes to that ceiling: MemHarness trained the competence into a 7B; this study rented it from a frontier model. Memory is rounding error on\nboth routes (ours +1.4 points at p = 0.5, theirs +2.2, and their own ablation shows raw replay *hurts* their trained policy), so lining the results up\ngives an ordering that is actor class all the way down: frontier, then trained 7B, then scaffolded 35B, then a frontier model on their plain\nscaffold-free harness (their strongest closed-model row, 62.1). Memory has no headroom left to buy here, the agentic twin of the head-to-head’s\n“hybrid ties a flat index” finding: the actor dominates, memory works the margin.\n\n### WebShop\n\n[WebShop](https://webshop-pnlp.github.io/) is product search against a 1,000-item catalog with a purchase at the end: 500 test sessions, a 15-step\nbudget. Two metrics bracket it: **score** grants partial credit for a near-miss purchase, and **SR** counts only perfect ones. The small catalog often\ncontains no exact match for the instruction, which caps attainable reward for every actor.\n\n#### Accuracy\n\n| Actor | Score | SR |\n|---|---|---|\n| Local 35B, no memory | 63.5 | 0.376 |\n| Local 35B + experience bank | 66.0 | 0.418 |\nFrontier actor (`claude-sonnet-5` ), no memory | 65.1 | 0.444 |\n| Frontier actor + experience bank | 65.2 | 0.450 |\n| MemHarness (GRPO-trained 7B) | 87.4 | 0.756 |\n\nWebShop delivers the agentic benchmarks’ only significant memory-ablation effect, and their clearest boundary ([Tab. 11](#table-11),\n[Fig. 15](#figure-15)). The weak actor gains 4.2 points of success rate from the bank (paired McNemar, p = 0.022), exactly where theory puts it: far\nfrom its ceiling, in a domain where episode know-how (query phrasing, option discipline, the scoring rules) transfers between tasks. The frontier\nactor gains +0.6 points of success rate (p = 0.8), landing at nearly the same total as the 35B; the trained policy’s 0.756 shows that what binds the\nuntrained arms is not the catalog alone but what untrained interaction with the hidden rubric extracts. Note what that ceiling does to actor class:\nthe same actor swap that buys 31 points of success rate on ALFWorld buys +1.6 score here, because WebShop’s reward is shaped by the catalog and its\npartial-credit mechanics, not by actor smarts. The deeper difference between the two benchmarks is what each one hides. ALFWorld states its goal in\nthe observation, so success yields to reasoning, and the one frontier actor tested gained nothing detectable from the bank there (0.959 against 0.973,\np = 0.5). WebShop grades with a rubric the agent never sees, weighing attributes, options, and price into partial credit over a catalog that often has\nno exact match, and no amount of reasoning over the observation reveals how that grader will score a near-miss. In these experiments nothing taught it\nexcept training against the reward signal itself, and retrieval never sees the reward: a bank stores what the agent did, not what the grader thought\nof it. That is why only training reaches the bar: every training-free arm lands at a score of 63 to 66 against MemHarness’s 87.4, and their number\ncomes from reinforcement learning against the environment’s own reward, which teaches the policy the reward’s *mechanics*, when to settle for a\npartial match, when to stop browsing, what an option is worth. Neither prompting nor a stronger actor replicates that, and none of the stores tested\nhere closes the gap from the outside.\n\nThat claim can be probed from the inside too, with one hard scope limit stated up front: the probe runs on my port of their frozen 7B actor, and that\nport falls well short of their published baselines, so it can speak about this port, not about their published system. Hold the port fixed on WebShop\n(full catalog, 500 sessions) and swap what its memory holds. Its own released 7,859-episode bank, injected through its own wire format and retrieval\nsemantics, scores 69.5 with a 0.306 success rate against the no-memory 71.0 and 0.300 (p = 0.69). The same bank under a different retrieval semantics,\nsituation-match instead of memory-text match, scores 69.1 and 0.298. On strict success the three arms are statistically indistinguishable; on\npartial-credit score both memory arms sit nominally *below* the no-memory baseline (by 1.5 and 1.8 points, the latter’s paired interval excluding\nzero), so if anything the bank costs this port a little. What that establishes: the tested injections did not help this port, so the bank does not\ncarry its value in a form that survives being bolted on from outside. What it cannot establish: that bank content or retrieval plays no role in the\npublished system, or that the training is the cause. The training remains the leading explanation because their own ablation points the same way from\nthe trained side, with raw replay *hurting* their trained policy, the same null as our frontier arms.\n\nLay the results beside each other and one pattern organises the realm: **memory paid only where the actor had headroom.** Two actor tiers on two tasks\ncannot establish a law, and the weak-versus-frontier difference in memory benefit is itself not significant; but every observation lines up the same\nway. A weak actor far from ceiling gains real points from retrieved experience; a frontier actor gains nothing detectable, having already reached what\nuntrained interaction with the task seems to extract (the trained policy’s higher bar shows the task itself is not saturated); a policy trained for\nthe task is actively hurt by raw replay. This is the consolidation null from earlier in the post seen from the other side: there a capable reader\nabsorbed the store’s disorder and left tidying nothing to buy, here a capable actor absorbs the task and leaves memory nothing to buy. Training buys\nits bar at the price of narrowness, too: served frozen outside its own harness, the released MemHarness model is acutely sensitive to exact prompt\nformat, the specialization reinforcement learning produces, where a frontier actor’s robustness is precisely the thing you rent. That places every\nretrieval-shaped system in this post, files, stores, temporal graphs, and hybrids alike, on one side of a line: bolt-on memory, model-agnostic, paid\nfor at write and read time, its value floating on the gap between the actor and the task. MemHarness sits on the other side: memory as trained\nbehavior, paid for in training compute, inseparable from its actor. The conversational benchmarks reward the first kind everywhere; the agentic\nbenchmarks reward it only while the actor is weak; past that line the question stops being “which store” and becomes “whose weights”.\n\nOne more disclosure belongs with the bar itself, because it cuts against the comparison. MemHarness’s numbers above are quoted from its paper. Running\ntheir *released* model on my own serving stack, under a faithful port of their harness and prompts, does not reach them: 0.581 macro on ALFWorld\nagainst their 0.830 without memory, and 71.0 score with a 0.300 success rate on WebShop against their 87.4 and 0.756. The shortfall has the same\nsignature on both benchmarks: the approach reproduces and the precision does not, exact option matches on WebShop and multi-step thermal sequences on\nALFWorld, a shape consistent with serving numerics and 8-bit quantization of a sharply peaked policy rather than with anything about memory, though\nthat attribution is untested (the separating run, bf16 against 8-bit, was not made). I report it because it makes the comparison’s frame explicit.\nTheir published bar stands as published; my arms are measured on my stack; and the distance between those two statements is the same cross-stack\ncaution this post applies to every other number it does not own.\n\nProvenance, disclosed: WebShop’s official dataset is org-locked, so the runs used the community mirror of the same files (1,000-product setting); all frozen arms are single runs, and the memory ablations are paired per-episode. The frontier arms cost about $22 of API spend on ALFWorld and $34 on WebShop, the latter including two voided protocol iterations.\n\n## Takeaways\n\n**The store architectures are bets about where memory’s cost sits, and each is right somewhere.** File-based memory bets on the model’s judgment and a filesystem, pays in model tokens at write time and literal search at read time, and buys transparency and simplicity: the right bet when memory is small, human-owned, and secondary to the task. Structured memory bets on an embedder and a ranker, pays in infrastructure, and is built for recall that holds up as history grows; the long-haystack pair here is consistent with that bet, though no scaling curve was measured.**When memory is the task, structure wins on both axes at once.** Under a fixed model on a hard benchmark, the structured store beat files by**28.7 points on held-out questions**(95% CI [22.1, 35.4]) at a fraction of the measured model tokens per correct answer.** Sparse memory abstains for free.**File-based memory wins the questions whose right answer is “I don’t know”, on both benchmarks: remembering less means over-answering less. Build the structured kind and you must budget for an abstention discipline.**Raw dated facts beat LLM-distilled graphs, and cost less twice over.** Inside the structured family, a good ranker over raw facts beat the graph lineage on the benchmark the graph is sold on, while the hosted graph spent six times the reader context to score lower. Whether place-plus-time beats a flat ranked query depends on where you ask: no detectable advantage on LoCoMo, a significant one on LongMemEval-M’s long haystacks, and no scaling curve connecting the two, which differ in more than history length.**Reasoning at ingest is a product decision, not an implementation detail.** A graph store spends several model calls on every message a user ever sends, where a raw-turn store spends one embedding: roughly two orders of magnitude more, about $14 to ingest one long history against about $0.03. It parallelizes, so it is a bill rather than a wall, but the bill scales with everything your users ever said. It is also why this study has no graph row on the long-haystack benchmark: that one row would have cost about 600 single-stream GPU-days of compute, or about $7,000 of hosted inference, to fill.**Consolidation is the third path, and the measured result is a null.** Both store architectures can run a background pass that reorganizes what is already stored, promoting what gets used or merging what repeats. Built here, it merged real duplicates and bought no accuracy at the scale tested. The design logic says it should pay once the mess outgrows what the reader can hold; that remains a hypothesis, since no consolidation-positive regime was measured.**The ruler can outweigh the architecture, and which one dominates depends on the benchmark.** Swapping the reader-and-judge stack moved a score by 6.9 points on byte-identical retrieval, more than the gaps between the three stores that work where they tie (0.3 to 3.6 points); on the long haystack the same store pair separates by 15. No single benchmark ranks these systems, and no number means anything without its protocol attached.**Retrieved memory paid only where the actor had headroom.** The agentic benchmarks draw the sharpest boundary in this post, as an observed pattern on two actor tiers and two tasks rather than a law: real points under a weak actor, no detectable effect under a frontier one, harm under a policy trained for the task. Where the task yields to reasoning, a frontier actor reaches the trained bar with no memory at all; where the reward has a structure only practice teaches, training stands alone, and it buys that bar at the price of narrowness. Below that line the architectures in this post are the game; at the line, the game becomes training.\n\nBoth headline numbers came from the same model answering the same questions. The only thing that changed was the shape of what it remembered with.\n\n## Appendix: methods and caveats\n\n*The story is complete without this section. What follows is the fine print: how the questions were split, every limitation I know about, and the\ncaveats each number carries. Nothing here overturns a result, but it tells you how far each one can be trusted.*\n\n### How the questions were split, and what “tuning” means\n\nThe 500 LongMemEval questions were split once, before any runs: a seeded, stratified 144-question tuning set, a 256-question holdout, and 100 left in reserve. Tuning-set numbers are provisional and never the headline; the holdout is scored exactly once per configuration. “Tuning” means prompts and configuration only, the model’s weights never move: the shared answering discipline (a six-round tool budget, an output cap, a truncation-rescue step, a facts-then-dates-then-verify format), the structured arm’s retrieval depth (top-12) and question-blind recency preload, and the file arm’s index budget (200 lines or 25 KB) and recall tools. The file-based arm got an equal tuning budget, and two of its frozen settings came from watching its own failures: the same truncation rescue the other arm has, and a guard against index-wiping writes. The holdout then judged both arms with everything frozen.\n\n### Limitations\n\n**The shared answering prompt has a heritage.** Its discipline was developed in earlier work whose read path resembled the structured arm’s. Equal tuning budget in this study is not equal ancestry; a file-native prompt built from scratch might serve that arm better, and I did not build one.**The file-based arm is a reconstruction, deliberately.** Its curation instructions are a faithful-as-documented rewrite of an unpublished original, every mechanism decision traced to a cited public source in the released spec. I chose not to validate against the shipping CLI: a closed-product run is a snapshot of whichever model version shipped that week, unreproducible by design. The fidelity burden is met by the traceable spec, the oracle control below, and a published envelope check (save rates, index shapes, read patterns) that anyone with the real tool can run to falsify the reconstruction.**One model family judges itself, and an independent judge has now audited part of it.** In the main experiment the judge is identical across arms and finished with a deterministic refusal pass, but it shares a model family with the arms it scores. For the store head-to-head, that concern has been tested: a frontier judge from a different vendor re-scored every arm’s published responses on a frozen 100-row sample. It grades uniformly stricter (4 to 7 points on every arm, agreement 0.91 to 0.96, Cohen’s kappa 0.82 to 0.89, with a 0.05 agreement spread that bounds agreement but cannot rule out directionally different errors per arm), and every ranking survives, with the hybrid’s win over the graph vendor significant on a tenth of the data (p = 0.024). The supported claim is rankings preserved on that frozen sample. The main experiment’s own judge remains unaudited.**The holdout was widened once, and the verdict stayed on the blind set.** Mid-study I extended the held-out set from 256 to all 356 non-tuning questions; the added questions had never been run or tuned on, but one arm’s solo holdout score had been seen, which is partial peeking. An independent review called it, so the verdict was locked on the blind 256. The tables in this post report all 356, because that is the set whose per-question rows are published, and widening changed the gap by 0.0002, so the two agree to the fourth decimal. The commit history proves the ordering.**Truncation hit the arms unequally**(20 of 144 file-based answers against 3, under a busy serving layer). The frozen rule counts both as wrong symmetrically, and the gap survives excluding every truncated row.**Absolute numbers reflect a minimal harness.** Both arms run a deliberately minimal shared loop because mechanism isolation is the point. A richer loop can help or hurt; for calibration, one production configuration of the structured mechanism reaches the mid-0.80s on this benchmark’s held-out questions.**The oracle control splits the gap descriptively, not causally.** Answering with the file arm’s*entire*memory directory in context and no tools scores 57.1% held-out, between the file arm’s 44.9% and the structured arm’s 73.6%: in aggregate, 12.2 points of the gap sit on the read side (saved but not found) and 16.5 on the write side (never written down). Per question the interventions are non-monotonic (the oracle also changes how the reader sees the store, and it loses questions the file arm won), so this is an aggregate split between arms, not an identification of where each point was lost. On preference questions the oracle beats the structured arm (72.2% against 61.1%), consistent with a search miss; on abstention it scores exactly what the file arm scores, which suggests over-answering tracks eager retrieval rather than context volume.**The comparison is measured at one history length.** Both arms ran at roughly 47 sessions per question. The 500-session variant is another two to three days of runs at ten times the history, outside this study’s window, so the prediction that files fall further behind as histories grow ([Tab. 3](#table-3)) is untested rather than confirmed. The long-haystack rows that do exist compare two structured stores, not files against structure.**Wall-clock is indicative, not controlled**; token counts are the load-independent cost metric.** Reproducibility has a scope.**The baseline, file arm, judge, and analysis run against any OpenAI-compatible endpoint; the structured arm calls a memory service, and its raw per-question rows are published for inspection either way in[a40-labs/memory](https://github.com/a40-labs/memory). The protocol, including every amendment and its timing, was recorded in the study’s private repository before the scored runs; the public repository carries the artifacts and an explicit ledger of what it cannot substantiate, not a timestamped registration.\n\n### Reading LoCoMo\n\n**Retrieval recall is not answer accuracy**, and LoCoMo numbers in the wild mix the two: a store can be scored on whether the right item merely\nsurfaces in its top-k, or the whole system on whether it answers correctly end-to-end. The gap is structural; this study’s store surfaces the right\nsession about 95% of the time on LongMemEval while its end-to-end accuracy sits at 0.73, the answering step consuming the rest.\n[MemPalace](https://github.com/mempalace/mempalace) publishes *only* retrieval recall on LoCoMo, with an explicit no-QA disclaimer, while the QA\nnumbers people quote (Zep’s disputed 84 / 75.14 / 58.44, mem0’s ~67) are LLM-judged answer accuracy from entirely different systems; putting one next\nto the other is comparing different sports. Beyond that, the benchmark is contested ground with no agreed state of the art and no clean third-party\nreproduction of the leaders, and the research community has argued it is close to saturated. Its questions cluster inside 10 conversations, so honest\nconfidence intervals must resample conversations, not questions. And its adversarial category inverts the incentive, rewarding refusal and punishing\nexactly the eager retrieval that helps everywhere else, which is why every LoCoMo score in this study is published under both scopes. One disclosure:\nthe file-based arm’s LoCoMo run predates a serving-layer retry fix, and 18 of its 300 answers died to output truncation and count as wrong; its\nnumbers are floors.\n\n### Head-to-head boundaries\n\nEvery head-to-head row is a single run, and identical reruns at temperature 0 drifted by 0.3 to 0.4 points, so the last decimal is noise. Graphiti-OSS\nis a best-effort parity configuration of the vendor’s open-source engine, not their hosted product. The hybrid’s retrieval was verified against a call\nledger to confirm its reranker actually ran on every row, because that component fails open (a degraded run returns a full, silently unranked result\nset; 295 early rows failed exactly that check and were quarantined and re-run). What is published, stated precisely: per-question verdicts, hit\ncounts, and context sizes for every head-to-head arm, with the shared judge prompt, in [a40-labs/memory](https://github.com/a40-labs/memory), where a\nverifier re-tallies every score and re-derives every statistic. The retrieved contexts, answers, and grades themselves are not republished: Zep\nCloud’s contexts are their data ([their repository](https://github.com/getzep/zep-papers) has them), and the full bundles for the other arms live in\nthe study archive. The published artifacts support re-tallying and re-deriving, not independent re-judging.\n\n### A side note for Obsidian users\n\nMany people wire an [Obsidian](https://obsidian.md/) vault (markdown notes joined by `[[wikilinks]]`\n\n) into an agent as its memory, usually over MCP;\n[Basic Memory](https://github.com/basicmachines-co/basic-memory) is the clearest example. By this post’s taxonomy that is the **file-based** kind:\nstorage the model curates, read back by text search. What the vault adds is an explicit link graph, but authored at write time by hand or by the model\n(one more thing curation has to get right) and walked deterministically rather than ranked: a graph doing a retriever’s job through foresight. The\nvault crosses toward the structured kind only when an embedding index is bolted on\n([Smart Connections](https://github.com/brianpetro/obsidian-smart-connections), Obsidian Copilot), which swaps the grep for exactly the ranked read\npath the structured store uses while the storage stays plain markdown. The honest picture is a spectrum: a plain vault sits with the file arm, an\nembedded vault sits closer to the structured one, and the write-side curation cost this post measured is paid the whole way across, right up until you\nstop asking a model to decide what to keep.", "url": "https://wpnews.pro/news/show-hn-i-evaluated-file-vector-graph-and-rl-based-memory-frameworks", "canonical_source": "https://www.pinglin.tw/blog/the-shapes-of-agent-memory", "published_at": "2026-08-15 14:23:15+00:00", "updated_at": "2026-08-15 14:40:59.215084+00:00", "lang": "en", "topics": ["artificial-intelligence", "machine-learning", "ai-agents", "ai-research"], "entities": ["OpenClaw", "MemHarness", "Claude Code", "Cline", "Cursor", "Windsurf", "mem0", "Letta"], "alternates": {"html": "https://wpnews.pro/news/show-hn-i-evaluated-file-vector-graph-and-rl-based-memory-frameworks", "markdown": "https://wpnews.pro/news/show-hn-i-evaluated-file-vector-graph-and-rl-based-memory-frameworks.md", "text": "https://wpnews.pro/news/show-hn-i-evaluated-file-vector-graph-and-rl-based-memory-frameworks.txt", "jsonld": "https://wpnews.pro/news/show-hn-i-evaluated-file-vector-graph-and-rl-based-memory-frameworks.jsonld"}}