# Show HN: I evaluated file, vector, graph and RL based memory frameworks

> Source: <https://www.pinglin.tw/blog/the-shapes-of-agent-memory>
> Published: 2026-08-15 14:23:15+00:00

# The Shapes of Agent Memory – Files, Stores, and Experience

August 12, 2026

An agent that remembers across sessions can keep its memory as curated markdown files, as an auto-mined structured store, or as trained experience. I measured all of them: files against a structured store under one fixed model, a store-only head-to-head across the structured lineages, and an experience bank on the agentic benchmarks where the state of the art trains memory into the weights.

Three side-by-side memory shapes: a file-based index of markdown lines, a structured store of embedded units linked by a graph, and trajectories of agent experience with one successful episode ringed.

An agent that only remembers within one conversation is a stranger with excellent manners: it greets you warmly every single day, and it has no idea
who you are. The moment you want it to know your projects, your preferences, and the thing you told it last Tuesday, you need memory that outlives the
context window. There are three common shapes a modern agent memory system takes ([Fig. 1](#figure-1)). Two are stores that sit beside a frozen model,
and they anchor opposite ends of a design axis; the third moves the memory behavior into the model itself.

The first keeps memory as **files the model curates**: a short index plus topic files, written in plain markdown, read back by searching and reading
them like any other file. It is what a coding agent reaches for when it has a filesystem and no database, and it is what
[Claude Code](https://code.claude.com/docs/en/memory), [Cline](https://docs.cline.bot/best-practices/memory-bank),
[Cursor](https://docs.cursor.com/context/memories), and [Windsurf](https://docs.windsurf.com/windsurf/cascade/memories) ship today.
[OpenClaw](https://docs.openclaw.ai/concepts/memory) is the most thoroughly worked-out version of it: its default `memory-core`

plugin keeps a curated
`MEMORY.md`

beside dated session logs, and it adds a background consolidation pass that the other file-based products do not have. The second keeps
memory as a **structured store**: every turn is mined into small atomic facts, embedded into a vector index, threaded into a temporal graph, and read
back by ranked retrieval. It is what you build when memory is the product, and it is what the dedicated memory startups
[mem0](https://mem0.ai/research), [Letta](https://www.letta.com/blog/agent-memory/), and [Zep](https://arxiv.org/abs/2501.13956) sell. The third keeps
memory as **experience the model is trained to use**: episodes still land in a bank, but everything that makes them memory, what to retrieve, whether
to trust it, how to turn it into action, is trained into the acting policy by reinforcement learning. It is the agentic state of the art
([MemHarness](https://github.com/KnowledgeXLab/MemHarness)), the shape the field reaches for when retrieval stops paying, and it is where this post
ends.

In this post, I measure which shape is better rather than argue it, which meant building the first two. The structured arm is a [hybrid](#hybrid) of
the [two structured lineages](#two-lineages-place-and-entity-and-time), plus a layer neither has: an associative graph learned from which places
actually get retrieved together, so recall can reach an item the query never ranked. The [file-based arm](#file-based) is a reconstruction of a
shipping coding agent’s auto-memory, traced claim by claim to public documentation and
[published with its spec](https://github.com/a40-labs/memory/tree/main/systems/file-based), not a strawman written to lose. Both run behind the *same*
agent loop, on the *same* local open-weight model, scored by the *same* judge on the same public benchmark, so only the memory layer can move the
number, and the per-question rows, with the scripts that recompute each figure and an explicit ledger of the few published scores whose rows could not
be released, are in [a40-labs/memory](https://github.com/a40-labs/memory). Hosted models come in where fairness demands: the
[head-to-head](#the-lineages-head-to-head) reads every store through one shared reader and judge, `gpt-4o-mini`

, the same model the graph vendor’s own
numbers were scored with; the agentic experiment fields a frontier actor, `claude-sonnet-5`

. The [trained shape](#experience-architecture) cannot join
the controlled comparison at all, because the training *is* the method: unplug its bank and you have a different policy, not a baseline. The
[last section](#remembering-what-worked-the-agentic-benchmarks) meets it on its home ground instead.

**TL;DR** The structured store beats files on accuracy and on token cost at once; files win where memory stays small, or where the right answer is “I
don’t know”. Against my own interest, the hybrid is statistically indistinguishable from a plain vector index on LoCoMo; paired on long-haystack
LongMemEval-M the same two stores separate by 15 significant points in the hybrid’s favour, which is consistent with structure paying as histories
grow (the arms differ in more than structure, so the bundle is what is measured), and no single benchmark ranks memory systems. Swapping the model
stack that reads and judges the memory moves the score further than swapping between any two of the stores that work, which is why numbers do not
travel between protocols. On the agentic benchmarks, retrieved experience paid only where the actor was weak with headroom left; where the task yields
to reasoning, a frontier actor reaches the trained system’s bar with no memory at all, and where the reward has a shape only practice teaches, the
trained policy stands alone.

## Store architectures

Both sit beside a frozen model and persist facts across sessions. The difference that matters is *who does the work, and when*.

### File-based

File-based memory spends its budget at write time, through the model. After a turn, the model decides whether anything is worth keeping, and if so it
edits a file: a new line in the index, or a paragraph in a topic file. The index is small on purpose, because it is loaded into context every session;
a common budget is the first 200 lines or so. Everything else lives in topic files that are *not* loaded until the model goes and reads them. Recall
is therefore whatever the model can find by keeping the index in view and grepping the rest. There is no embedder and no ranker. The whole system is
the model’s own judgment plus a text search, which is exactly why it is so easy to ship: if you have file tools, you have this.

This approach is everywhere in shipping coding agents: Claude Code’s auto-memory (a per-project `MEMORY.md`

index over model-curated topic files), the
[memory tool](https://platform.claude.com/docs/en/agents-and-tools/tool-use/memory-tool) primitive in Anthropic’s API, the community’s
[Cline “Memory Bank”](https://docs.cline.bot/best-practices/memory-bank) and its descendants, and the automatic memories in Cursor and Windsurf. Keep
it distinct from the *instruction*-file family (`AGENTS.md`

, `CLAUDE.md`

, `.cursorrules`

), which is human-authored static context; the accumulated,
model-written kind is what this post evaluates.

### Structured

Structured memory spends its budget at write time too, but not through the model. Every turn is mined into atomic units, each embedded and stored with dense and sparse vectors, and salient facts are threaded into a graph whose edges carry validity windows so a later fact can supersede an earlier one. A background pass consolidates duplicates and merges the graph. Nothing on the write path asks the model to reason; it is extraction and embedding. Recall is a ranked hybrid query, and the most salient units are preloaded before the first user word, so the agent often answers without searching at all.

This is what the dedicated memory startups sell: [mem0](https://mem0.ai/research) extracts facts into a vector-first store with an optional graph
layer, [Letta](https://www.letta.com/blog/agent-memory/) (formerly MemGPT) pages tiered memory in and out of context, and
[Zep](https://arxiv.org/abs/2501.13956) builds a bi-temporal knowledge graph, the strongest form of the idea. All of them put an extractor and a
ranker where the file-based approach puts the model’s judgment and a grep.

Two design decisions fall straight out of this split, and they are the whole game:

**What gets saved.** File-based memory saves what the model*chooses*to save. Structured memory saves*everything*, then ranks.**What gets found.** File-based memory finds what a literal search surfaces from an index that must stay small. Structured memory finds what a similarity ranker surfaces from a store that can grow without bound.

#### Two lineages: place, and entity-and-time

Calling all of that one “shape” hides a real split, because the structured pole has two lineages that organize memory around opposite primitives: **by
place, or by entity and time.**

The place-organized lineage files memory by *where it belongs* rather than by who it is about. [MemPalace](https://github.com/mempalace/mempalace) is
the cleanest public example: people and projects become wings, topics become rooms, and the original conversation text lives in drawers inside them,
retrieved by semantic search scoped to a region rather than swept across a flat corpus. The defining choice is what it declines to do at write time.
It stores the text verbatim, and does not summarize, extract, or paraphrase, so ingest is an embedding and a filing decision with no model reasoning
in it at all. Its structural weakness is aggregation: an answer scattered across many rooms depends on one ranked query surfacing all of it at once,
and no artifact in the store has gathered it in advance.

The entity-and-time lineage (Zep’s [Graphiti](https://github.com/getzep/graphiti)) stores a knowledge graph instead: raw messages kept as ground
truth, LLM-extracted entity nodes that are resolved and deduplicated across sessions and carry maintained summaries, and one-line distilled fact edges
between entities. Every edge holds validity timestamps, relative dates are resolved to absolute ones at ingest, and a contradicting new fact *closes
the old edge’s validity window* rather than deleting it. The reader receives distilled facts with date ranges plus entity summaries, never raw
messages. The cost inverts: an LLM reasons at ingest, on every message, for extraction, resolution, and invalidation.

Three consequences follow ([Fig. 3](#figure-3)):

**Aggregation.** An entity node accumulates every fact about a person*by construction*, so an enumeration question (“what are all of X’s hobbies?”) arrives with a pre-built aggregate. A place-organized store has no such artifact: it must hope one ranked query surfaces every scattered item, and the members of an enumerable answer are usually semantically far apart (running, pottery, and movie nights share little beyond the person), so no single query ranks them all into the top-k even when every item is in the store. Keep this weakness in mind for the results: the hardest questions for this study’s structured arm, on both benchmarks, are exactly the ones that assemble an answer from facts scattered across many sessions.**Time.** Supersession lives in the store for one lineage (validity windows the reader can trust) and in ranking heuristics for the other.**Cost.** Place-organized is cheap at write time and leans on retrieval; entity-and-time pays heavy LLM cost at ingest to make reading cheap and precise.

Neither dominates. If your workload is scoped recall over evolving topics, the place lineage’s load layers are the better fit; if it is cross-session aggregation and “what is true now”, the graph lineage earns its ingest bill. Which raises the obvious question: why not take the cheap half of each?

### Hybrid

The hybrid takes exactly that bargain, and it is the design this study measures ([Fig. 2](#figure-2)’s right panel is its architecture). Take the
place-organized store’s write path wholesale: atomic dated facts filed by location, embedded with no LLM reasoning, recalled by layered loads and
ranked hybrid search. Then borrow one thing from the entity-and-time lineage: validity windows, so a contradicting new fact can close an old one’s
window instead of competing with it at recall time. That combination is not unique to this study, and it would be misleading to imply otherwise:
MemPalace ships a temporal entity graph with validity windows of its own, alongside its rooms. What differs here is smaller and more specific, and it
is the third layer below.

What the hybrid buys is the cost profile of place with the time semantics of the graph: writes stay embedder-cheap, and “what is true now” questions
get dated, supersedable facts rather than ranking heuristics alone. One temporal nuance the graph lineage does not spell out: supersession must
distinguish conflicting *states* from additive *events*. “Works at Acme” should close “works at Beta”; “scored 2 goals this week” must never close
“scored 3 goals last week”, or counting questions become unanswerable. What the hybrid deliberately leaves out is the graph lineage’s expensive half:
no LLM at ingest, so no entity resolution and no maintained summaries, and cross-session aggregation stays its structural weak point.

The design’s answer to that weak point is a third layer the two lineages do not have: an **associative graph learned from usage statistics**.
Locations that co-occur in retrievals more often than chance predicts get linked (a statistical test, not embedding similarity: “these go together” is
a stronger claim than “these look alike”), and recall runs as *anchor, expand, fuse*: ranked search anchors on what it can find, the association graph
expands to linked locations the query never ranked, and the fused result caps the graph’s contribution ([Fig. 4](#figure-4)). The design principle
underneath answers both lineages’ disclosed failure modes at once: **the graph is only ever allowed to add candidates, never to displace the anchor
set.** That guards the retrieval, not the reader, since added context can still distract the model downstream; what the measurements showed is no harm
at the scale tested, not monotonicity. Where the entity-and-time store is hard-bounded by its extractor, the hybrid keeps raw dated facts as the
anchor, so a graph failure degrades to plain ranked retrieval; and where pure place organization can never *reach* an item its one query failed to
rank, expansion gives it a query-independent path there.

One scope note for honesty, and it is a large one: the benchmark protocol below writes facts directly into the store, which never triggers the background consolidation where supersession lives (the next subsection is about that pass), and the association graph starts empty. So the configuration actually measured is the hybrid’s place-organized core: dated facts plus hybrid ranked search. Read its scores as a floor, with one update from later work that cuts against my own design: when the association graph was subsequently seeded from real co-retrieval statistics and given one controlled, paired shot at exactly the aggregation failures it exists to fix, it changed nothing. No harm (the capped fusion held), but no recovery either. The associative layer stays a design capability, not a measured contributor.

### Consolidation

Every architecture so far has been described by two paths, write and read, and every description is incomplete. There is a third path, and it is not a
fourth architecture: a pass that runs between sessions and reorganizes what is already stored. The field calls it **dreaming**, after the
consolidation that happens in sleep, and it is the only path that can repair a store that is already wrong.

It cuts across the taxonomy rather than extending it. Files can be consolidated, place-organized stores can be consolidated, and the entity-and-time lineage consolidates so eagerly it is easy to miss: resolving a mention against existing entities and closing a superseded fact’s window is exactly this work, moved to ingest and paid per message. That relocation is the real choice on offer. Consolidate at ingest and every write pays for order the store may never need; consolidate in the background and writes stay cheap while the store carries its own mess until the pass comes around.

The two implementations below sit at opposite ends of that trade ([Fig. 5](#figure-5)).

[OpenClaw](https://docs.openclaw.ai/concepts/dreaming) runs the file-based version nightly, in three phases borrowed from a night’s sleep: a light
phase deduplicates the recent buffer, a phase named after REM (the rapid-eye-movement stage where human brains replay the day and connect it to older
memories) looks across conversations for recurring themes, and a deep phase promotes survivors into `MEMORY.md`

, the index every session loads. What
makes it more than cleanup is the gate: an item earns its place by being *used*, clearing a score threshold and several recalls across distinct
queries. The store learns what matters from what the agent kept reaching for, a signal neither the write nor the read path can see.

This study’s hybrid does the same job on different material, in three tiers. Identical writes never duplicate, because a unit’s content hash is its
primary key. A minute after a conversation goes idle, a pass rebuilds that conversation’s index cards. Then once a day the whole store is swept: units
are clustered by similarity, using the vectors they already carry, and each cluster collapses to its longest phrasing with the others’ provenance
folded in. The bar is set high so merely *related* facts stay apart, and because the pass reuses stored vectors instead of re-embedding, it spends no
model calls at all. So OpenClaw promotes upward into a file the model reads; the hybrid collapses sideways into a store the ranker searches.

Two limits come with that design, and both are visible before any measurement. Cheap similarity catches restatements but misses the same event told in other words, and closing that gap means a model judging each pair, which puts per-item reasoning cost back into the one path that had none. And the value of tidying at all depends on the reader: a model answers correctly about a handful of restatements sitting in front of it, merged or not, so storage-level cleanup earns its keep only once duplicates outnumber what the reader can hold. Consolidation pays most on long histories read by weak models, least on short ones read by strong ones. The measurements here cover only the second regime: run at roughly fifty sessions per history it merged real duplicates and bought no accuracy, and the main experiment never triggers it at all, so consolidation contributed nothing to the structured arm’s scored numbers; whether running it would raise or lower them is unmeasured.

## Experience architecture

Every store architecture above shares one assumption so basic it is easy to miss: **the model that uses the memory is frozen.** The store gets
smarter, better ranking, better structure, better time semantics; the reader of it does not. The agentic memory line of work drops exactly that
assumption, and [MemHarness](https://github.com/KnowledgeXLab/MemHarness) ([paper](https://arxiv.org/abs/2607.28272)) is its cleanest current example:
hold the store simple, and train the model’s *use* of it instead.

The bank half looks deliberately familiar ([Fig. 6](#figure-6)). After every episode the trajectory is summarized and written into a vector store with
semantic embeddings, deduplicated semantically, and periodically pruned by empirical utility, so entries that keep paying rent stay. By this post’s
taxonomy that is a structured store: inspectable, swappable, nothing a reader of the sections above has not seen.

The difference is everything downstream of retrieval. Where a store architecture hands retrieved items to a frozen model and hopes its judgment
suffices, the experience architecture makes that judgment the trained object. Acting is a five-stage policy: **observe** the current state;
**retrieve** the top-k experiences, each paired with the source observation it was learned from; **critique** the retrieved experience against the
current state (does this actually apply here?); **reconstruct** it into state-specific guidance when it does, or reject it and fall back to
self-reasoning when it does not; then **act**. The whole pipeline is trained end-to-end with reinforcement learning (GRPO, group-relative policy
optimization over grouped rollouts, with format rewards that keep the retrieval and reconstruction stages from collapsing), cold-started from a couple
hundred teacher-written memory records.

Why go to that expense? Because the untrained alternative is not merely weaker, it is negative: in their own ablation, handing the trained policy raw
replayed episodes instead of reconstructions makes it *worse* (76.4 with no memory to 70.1 with raw replay). Retrieval gets the experience into view;
nothing about a frozen model guarantees the experience gets *used*, and a policy that has learned when to trust a memory and how to rewrite it for the
situation at hand is solving a problem that no amount of store engineering touches. The cost profile inverts accordingly: the store architectures pay
at write or read time and bolt onto any model; the trained one pays in training compute and is inseparable from the one model it trained.

That is also why it cannot join the controlled comparison that follows: unplug its bank and you have a different trained policy, not a baseline. The honest meeting point is its home ground, the agentic benchmarks, where the final section takes the structured store to meet it.

## Evaluation

Two public benchmarks carry the comparison: LongMemEval as the primary, LoCoMo as the second opinion. Each gets a subsection below, because what a benchmark measures, and what its numbers have been made to say in the wild, decides how much a score is worth.

Every result table below compares the same three arms, named the same way throughout. Each table and figure states the sample it was scored over,
written out in the tables and abbreviated as **n** in the figures, meaning the number of questions, games, or sessions behind that number:

**No-memory**: the same agent loop with the memory layer removed. The floor that sizes what memory contributes at all, and proof that the judge cannot be gamed by refusing everything.**File-based**: the markdown reconstruction described above. An LLM-curated index plus topic files, recalled by grep and read.** Structured**: the hybrid described above. Dated atomic facts, embedded on write with no LLM, recalled by ranked hybrid search.

The two pure structured lineages, place-organized and entity-and-time, do not run in the main experiment; they get their own store-only head-to-head at the end of this section, where the graph vendor’s production system competes through its own published retrieval.

The controls are the point. Both arms ran the same agent loop, the same locally served open-weight model as the answerer
([ Qwen3.6-35B-A3B-mxfp4](https://huggingface.co/mlx-community/Qwen3.6-35B-A3B-mxfp4)), the same embedder where one was needed, and the same judging
pipeline: a model judge scores each answer against the gold one, and a deterministic pass then re-classifies refusals (saying “I don’t know” counts as
correct only when the answer genuinely was not in the history). The judge is identical for both arms, which removes per-arm judge configuration as a
factor, though a shared judge can still prefer answer styles that correlate with an arm; it is also one model family scoring its own outputs. Both
concerns are flagged rather than hidden. The file-based arm is a faithful implementation of the documented approach (Claude Code’s auto-memory,
described above), with a couple of deviations that make it slightly

*more*robust than the standard, not less. Anywhere the two arms could differ for a reason other than the memory architecture, I held them equal.

One thing this is *not*: a measurement of any shipping product. I reimplemented the file-based *architecture* and drove it with a local open-weight
model, so these numbers say nothing about how Claude Code, Cline, or anyone else performs in their own product, on their own model. Naming products is
about where the architecture comes from, not a leaderboard of them. What is being compared is the memory architecture, with everything else held
fixed.

That fixity is the whole reason to bother. Published memory numbers are notoriously hard to compare across vendors, and the LoCoMo subsection below tells the canonical story. A number is only worth anything when you know what was held constant. Here, everything but the memory architecture was.

### LongMemEval

[LongMemEval](https://arxiv.org/abs/2410.10813) is a public suite for long-term conversational memory: 500 questions spanning categories that separate
the easy from the hard: single-session recall, multi-session joins, knowledge updates (“what is the *current* value”), temporal reasoning, and
abstention (knowing when the answer was never stated). Retrieval and answering both count: the system has to surface the right memory *and* answer
from it, and an LLM judge scores the answer against the gold one. It comes in two sizes: **LongMemEval-S**, where each question sits over a history of
roughly 47 prior sessions, and **LongMemEval-M**, the same questions over roughly 500-session haystacks. The main experiment runs -S; the -M numbers
close this subsection, because scale is exactly what they measure. It is the primary benchmark here because its haystacks are long enough to punish
weak recall and its categories name the exact failure modes the architectures should differ on.

Every headline number below is measured on the **held-out** questions, never the tuning split; both arms scored fractionally *higher* on questions
they had never seen, which is the opposite of what overfitting looks like. How the 500 questions were split, what “tuning” concretely means, and every
limitation a skeptic should weigh (prompt heritage, reconstruction fidelity, self-judging, and an oracle control that splits the gap descriptively
into read-side and write-side halves) are collected in the [appendix](#appendix-methods-and-caveats). The one that matters most mid-read: the
file-based arm got an equal tuning budget, and two of its frozen fixes came from watching its own failures.

#### Accuracy

| Category (held-out questions) | Structured | File-based |
|---|---|---|
| Temporal-reasoning (91) | 0.802 | 0.407 |
| Multi-session (97) | 0.608 | 0.330 |
| Knowledge-update (36) | 0.833 | 0.528 |
| Single-session-user (52) | 0.923 | 0.673 |
| Single-session-assistant (44) | 0.568 | 0.273 |
| Preference (18) | 0.611 | 0.333 |
| Abstention (18) | 0.778 | 0.889 |
Overall (category-reweighted) | 0.736 | 0.449 |

On the held-out questions ([Tab. 1](#table-1), [Fig. 7](#figure-7)), the structured arm (the hybrid) scored **73.6%** and file-based scored **44.9%**
(category-reweighted; raw 73.1% and 44.1%). The paired difference is **28.7 points**, 95% confidence interval **[22.1, 35.4]**, comfortably clear of
zero. Three checks say the result is solid rather than lucky: the number barely moved from the tuning set to the held-out set for either arm (both
actually ticked *up*, the opposite of an overfitting signature); widening the held-out set from 256 to all 356 non-tuning questions changed the gap by
0.0002; and throwing out every question where either arm’s answer was truncated by the serving layer still leaves 74.1% against 50.2%. For scale, a
no-memory baseline answering the same questions with no memory at all scores **9.8%**, so both architectures are doing real work; the question is how
much.

The category breakdown says *why*, and it is not subtle. The widest gap is **temporal reasoning** (80% against 41%), and **multi-session** (61%
against 33%) is close behind and clearest about the mechanism: its answers are assembled from facts mentioned in several different conversations. A
literal search over a deliberately small index is the wrong tool for that. If the joining fact sits in a topic file the model never thought to grep,
it is simply gone, and the model, to its credit, usually says it does not know rather than inventing an answer. Ranked retrieval over an unbounded
store does not have this failure mode: the fact was saved whether or not anyone predicted it would matter, and similarity, not a filename, brings it
back. **Knowledge updates** (83% against 53%) tell the same story from another angle.

There is exactly one category the file-based arm **wins**: **abstention** (88.9% against 77.8%), knowing that something was never said. That is not a
rounding artifact, and it replicates: on the second benchmark’s adversarial questions, in the LoCoMo section below, the file-based arm beats the
structured one by an even wider margin, and a no-memory baseline that refuses everything beats them both. The mechanism is the same in both places and
it is worth stating plainly, because it cuts against the headline: **a store that remembers less over-answers less.** Ranked retrieval almost always
surfaces something plausible enough to tempt an answer, while a curation-limited store often has nothing to offer and the model correctly says so.
Eager retrieval needs an abstention discipline bolted on; sparse memory gets one for free.

#### Cost

Accuracy is half the story. The other half is what each answer costs, and here file-based memory pays twice: more tokens, for a worse answer.

Comparing cost honestly requires separating two currencies. Both arms spend **model tokens** (prompt plus completion through the 35-billion-parameter
model, as the serving layer reports them; hidden reasoning is not always reported, so these are floors), and those are directly comparable. The
structured arm *additionally* spends **embedder tokens** (a 2-billion-parameter model producing vectors), which cost orders of magnitude less per
token and have no counterpart on the other side. Summing them into one number would be meaningless, so I never do; nor does this ledger price latency,
per-token rates, or infrastructure, so “cheaper” here means fewer measured model tokens, not a total cost of ownership.

| Chat tokens per question | File-based | Structured |
|---|---|---|
| Writing memory (LLM curation, amortized per question) | 246.1k | 0 (verified) |
| Answering (recall plus reasoning) | 40.4k | 19.3k |
Total | 286.5k | 19.3k |
Total per correct answer | 665k | 27k |
| Embedder tokens per question (estimated; separate currency) | 0 | 107.8k |
| Wall-clock per ~50-session ingest | ~35 min | ~5 min |

In model tokens ([Tab. 2](#table-2), [Fig. 8](#figure-8)), per question: file-based **287k** against structured **19k**. Divide by accuracy to get the
cost of a *correct* answer, which is what you actually pay for, and it is **665k against 27k**. On top of its 19k, the structured arm spends about
**108k embedder tokens** per question on the write path. Even charging those at par with model tokens, which wildly overstates them, it remains the
cheaper architecture. The reason is the write path ([Fig. 9](#figure-9)).

Curating a file is a *reasoning* act. For every session in a history, the model reads the current index, decides what is worth keeping, and rewrites a
line. Over a full ingest that came to roughly **246k model tokens per history** and about **35 minutes** of wall-clock per history on my hardware.
Structured memory writes by embedding, no model in the loop, which finished the same history in about **5 minutes**, roughly a sevenfold speedup on
the write path. The two write costs are in different currencies (one is LLM chat tokens, the other is embedding-model tokens), so I never subtract one
from the other, but the direction is not close.

The read paths differ too, in a way that compounds. File-based recall is iterative: keep the index in view, grep, read a file, maybe grep again, then answer. That longer, multi-round path also turned out to be more fragile. Under a busy serving layer the file-based arm hit truncation on 20 of 144 answers against the structured arm’s 3, precisely because it asks the model to generate more, over more rounds, with more chances to be cut off. Some of that is my serving setup, but part of it is intrinsic: a longer read path has more surface to fail on.

#### The long haystack: LongMemEval-M

The -M variant asks the same questions over roughly ten times the history. The two store-only rows in [Tab. 3](#table-3) ran on the same 100-question
sample, drawn once by seed before either arm ran (the drawn ids are published in the repo; the ordering of draw and runs rests on the study log), one
retrieval and one reader call each under the benchmark’s official per-category judging, so their comparison is paired even though neither is paired
with the main experiment:

| System (LongMemEval-M) | Score | Questions scored |
|---|---|---|
Hybrid (store-only) | 0.750 | 100 (pre-drawn sample) |
| Place-organized (MemPalace, store-only) | 0.600 | The same 100 |
| Hybrid (full agent loop) | 0.632 | 500 (complete set, different harness) |
| File-based | None | None: another ~2-3 days of runs at 10x the history |
| Entity-and-time (Graphiti OSS) | None | None: ~600 single-stream GPU-days, or ~$7,000, to ingest |

Long histories are the regime structure exists for, and the paired rows put a number on the claim: fifteen points, rescuing 22 questions against losing 7, exact p = 0.008 under the same paired test as the head-to-head below. The gap clears the sample’s own confidence interval, and the shape of the win matches the mechanism, with the hybrid sweeping the single-session categories (14/14 and 11/11) and pulling ahead on the multi-session and temporal content that long haystacks exist to test.

Store-only gave the reader one ranked context and outscored the full loop on -M (0.750 vs. 0.632) and the 100-question -S sample in [Tab. 7](#table-7)
(0.80 vs. 0.72). Prompt and loop effects were not isolated, so both gaps are directional.

Graphiti OSS is unscored because ingesting the benchmark would be prohibitively expensive: it requires a reasoning call for each of the haystack’s 3.7 million messages, while an embedder takes milliseconds. At the roughly 14 seconds per message measured on this hardware, that is about 600 days of single-stream GPU time; parallel serving divides the wall-clock but not the bill, and renting a small hosted model to do the same work would have cost roughly $7,000 at list prices. I was not willing to spend either on one row of one table, so the row stays empty and the reason is published.

The file-based row is empty for a simpler reason: runtime. Fifty questions over ten times the history is another two to three days of runs, which fell outside this study’s window. Those runs were the only test of what this post’s main comparison implies about scale — that files fall further behind as histories grow — so that claim is measured at roughly 47 sessions and untested at 500, precisely where I expected the gap to be widest. Untested is not refuted, and the long-haystack rows above are a different pair: they say nothing about how files would have done. My guess is that the gap widens rather than narrows, and mostly on the write side, since an index capped at a couple of hundred lines cannot grow tenfold with the history behind it — curation drops more of what was never written down, while a ranked store simply retrieves from a larger pool. A guess is all that is, though, and the run that would settle it remains outstanding.

That is not a knock on the lineage so much as a statement of what it costs to reach the regime that matters. -M is where real assistants drift, and the paired rows above show it is where the benchmarks disagree: the flat store that is indistinguishable from the hybrid on LoCoMo falls 15 points behind here, where multi-session organization starts to pay.

### LoCoMo

A single benchmark is a single opinion, so the same three arms (no memory, file-based, structured) also ran on
[LoCoMo](https://arxiv.org/abs/2402.17753), the other widely used long-term-conversation suite: 10 very long two-speaker conversations, each spanning
dozens of sessions, with 1,986 questions across single-hop, multi-hop, temporal, open-domain, and adversarial (unanswerable) categories.

LoCoMo needs its story told before its numbers can be trusted, because it is the benchmark on which the field’s most public scoring fight happened.
Zep reported 84% on it. mem0’s CTO [filed an issue against their evaluation code](https://github.com/getzep/zep-papers/issues/5) arguing the real
number was 58.44: the adversarial category had been counted in the numerator but excluded from the denominator, and the baseline configurations
differed. Zep’s [rebuttal](https://blog.getzep.com/lies-damn-lies-statistics-is-mem0-really-sota-in-agent-memory/) re-ran with the error fixed and
reported 75.14, while pointing back at mem0’s own reporting (whose LoCoMo figure has been cited
[at both 67% and 92.5%](https://arxiv.org/abs/2504.19413) depending on the write-up). One system, one benchmark, three published numbers spanning 25
points, and the memory architecture never changed: the swing came entirely from scoring conventions, judge choice, and which categories count.

So why keep the benchmark? Because the dispute indicts the reporting, not the questions; because it is the suite the vendors actually compete on, so
results on it travel; and because the fight teaches exactly this study’s premise, that a number means something only inside a fixed, published
protocol. The dispute constrains the protocol here in three ways. Every score is published under both scopes, with and without adversarial, because
whether to count that category is precisely the axis Zep and mem0 fought over. The analysis resamples whole conversations rather than questions,
because LoCoMo’s questions cluster inside just 10 conversations and pretending otherwise makes intervals too tight. And I evaluate a disclosed,
seeded, category-stratified sample of roughly 30 questions per conversation. The rest of the benchmark’s fine print (retrieval recall confused with
answer accuracy in the wild, saturation critiques) lives in the [appendix](#appendix-methods-and-caveats).

#### Accuracy

The run completed ([Tab. 4](#table-4), [Fig. 10](#figure-10)), and the result is the study’s most honest one, because the verdict depends on the scope
in exactly the way the dispute predicts:

| LoCoMo (300 questions, cluster CI) | No memory | File-based | Structured |
|---|---|---|---|
| All questions | 0.217 | 0.387 | 0.497 |
| Excluding adversarial | 0.017 | 0.356 | 0.561 |

Excluding adversarial (the scope the benchmark’s own convention arguably prescribes), the structured arm wins clearly: the paired difference is +0.205
with a cluster confidence interval of [+0.063, +0.356], and it dominates the memory categories (temporal 0.694 against 0.306, open-domain 0.656
against 0.410). Include adversarial and the verdict collapses to a statistical tie (+0.110, CI [-0.007, +0.240]), because the file-based arm abstains
better on unanswerable questions (0.508 against 0.246): its curation-limited store simply has less material to over-answer with, while ranked
retrieval almost always surfaces *something* plausible enough to tempt an answer. Even the no-memory baseline “wins” adversarial outright (1.000) by
refusing everything, which is why a blanket-refusal system still only scores 0.217 overall. The lesson generalizes: eager retrieval needs an
abstention discipline, and a store that remembers less over-answers less. Both readings are published; neither is smoothed away. One disclosure: the
file-based arm’s run predates a serving-layer retry fix, and 18 of its 300 answers died to output truncation and count as wrong under the symmetric
rule; its numbers are floors.

#### Cost

The cost asymmetry survives the second benchmark, at a smaller scale ([Tab. 5](#table-5), [Fig. 11](#figure-11)). LoCoMo’s conversations are far
shorter than the primary benchmark’s haystacks, so the file arm’s curation bill shrinks, but the ordering does not change: 22k model tokens per
question against the structured arm’s 12k, and 58k against 23k per correct answer, with each conversation’s ingest amortized over its sampled
questions. The structured write path again spends zero LLM tokens (verified against the serving ledger) plus about 0.8k embedder tokens per question
in its separate currency. The gap compressing from roughly fifteenfold to roughly twofold is itself the finding: write-time curation is priced by
history length, which is the primary benchmark’s cost story wearing smaller numbers.

| Chat tokens per question | File-based | Structured |
|---|---|---|
| Writing memory (amortized per question) | 4.9k | 0 (verified) |
| Answering (recall plus reasoning) | 17.6k | 11.6k |
Total | 22.4k | 11.6k |
Total per correct answer | 58k | 23k |
| Embedder tokens per question (separate currency) | 0 | 0.8k |

### The lineages, head-to-head

Everything above compares files against one structured design, and it leaves the lineage question open: inside the structured shape, does the graph
earn its ingest bill? The head-to-head answers it in the most controlled frame available: strip every system down to its retrieval and hold everything
else constant. Each store contributes exactly its top-20 results for the same 1,540 non-adversarial LoCoMo questions; one fixed reader (gpt-4o-mini)
answers from that context alone, one fixed judge scores it, and every store’s row is produced by the same script. The entity-and-time lineage appears
twice: as [Zep’s published retrieval contexts](https://github.com/getzep/zep-papers), their production system’s real output (and a fairness note they
are owed: their published 75.14 reproduces from their own artifacts; 0.7461 is the same context re-scored under this unified frame), and as their
open-source engine [Graphiti](https://github.com/getzep/graphiti) run end-to-end on their paper’s recipe, both embedders it names. These rows sit far
above the bare-loop table above because everything about the frame differs; they are comparable to each other and to nothing else. For orientation,
[Tab. 6](#table-6) puts the four methods side by side (pure place-organized differs from the hybrid only by dropping the temporal layer, so the hybrid
bounds it closely):

| Method | Write path | Read path | Time handling | Aggregation | Where measured |
|---|---|---|---|---|---|
| File-based | LLM curates markdown | Index in context + grep/read | None built in | Index + luck | Main experiment |
| Place-organized | Embed and file, no LLM | Layered loads + ranked search | Ranking heuristics | Ranked-query hope | This head-to-head |
| Entity-and-time (Graphiti) | LLM extracts, resolves, invalidates | Distilled facts + entity summaries | Validity windows in the store | Entity nodes, by construction | This head-to-head |
| Hybrid (place + time) | Embed and file, no LLM | Ranked search; associative expansion in the design | Dated facts; windows in the design | Ranked-query as measured | Main experiment and here |

#### Accuracy

| Store (one reader and one judge throughout) | LoCoMo (1,540 questions) | LongMemEval-S (100 questions) |
|---|---|---|
| Hybrid (this study’s structured arm) | 0.7825 | 0.80 † |
| Place-organized (MemPalace, dense over raw turns) | 0.7792 | 0.60 † |
| Entity-and-time (Zep Cloud, published contexts) | 0.7461 | None |
| Entity-and-time (Graphiti OSS, two embedder configs) | 0.5338 / 0.5286 | 0.35 |

Four findings ([Tab. 7](#table-7), [Fig. 12](#figure-12)).

**The hybrid beats the graph vendor’s published retrieval**, and the difference is real under a paired test ([McNemar’s test](https://en.wikipedia.org/wiki/McNemar%27s_test), which scores only the questions the two systems disagree on; p < 0.01). So, separately, does the plain dense store: raw turns beat distilled facts here before any hybrid machinery is added at all.**The hybrid is statistically indistinguishable from the flat vector index**(+0.3 points, paired CI95 [-1.8, +2.4]), which is the uncomfortable finding and belongs in the open: no detectable gain from place-plus-time on this benchmark, though the interval allows small effects either way. It is also only half the story: the same two stores, paired on LongMemEval-M’s long haystacks ([Tab. 3](#table-3)), separate by 15 points (0.750 against 0.600, p = 0.008). The two arms differ in more than structure (fusion, reranking, recency, atomic facts against raw turns), so read that as this implementation beating that one at long histories while being indistinguishable from it at short ones: the benchmark-disagreement point again, made by one pair of systems.**The distilled pipelines lose to the raw-text pipelines.** Both graph rows hand the reader LLM-distilled facts and entity summaries; both trail every raw-turn store, and the strongest graph row loses 3.3 points to the flat index while spending six times its context. The pre-built entity aggregates that make the lineage attractive for enumeration questions do not surface as a net win anywhere in this table; whatever they recover, the distillation loses more elsewhere. This is the single-session regression Zep itself discloses, visible benchmark-wide once the reader is held constant.**An LLM-at-ingest design is bounded by its extractor, and the extractor bill is real.** Graphiti OSS with the vendor’s own models lands around 0.53, so much of its 21-point gap to the vendor’s cloud contexts sits outside the extraction model’s capability. Driven instead by this study’s local 35-billion-parameter model, the same engine collapses to 0.29 (a study-log run whose per-question rows are not among the published data), and the mechanism is visible at ingest: it extracts several times fewer facts per message than the cloud output implies. A store that only knows what its extractor wrote down starves quietly.

The LongMemEval-S column of [Tab. 7](#table-7) is the same three stores under the benchmark’s official per-category rubric on a pre-drawn 100-question
sample (the hybrid’s 0.80 there is a single retrieval and a single read under a shared reader prompt, which is why it sits above the same system’s
agent-loop 0.72 on this sample; [Tab. 3](#table-3)’s discussion unpacks that ordering). Read the two columns across and the point makes itself: the
flat store that ties the hybrid on LoCoMo trails it by 20 points on LongMemEval, because LoCoMo mostly rewards verbatim lookup inside a few dozen
sessions while LongMemEval forces multi-session organization. The two benchmarks disagree about the same pair of systems, in opposite directions. No
single benchmark ranks memory systems.

The same frame also measures the thing this post keeps insisting on, and it is worth putting a number on rather than gesturing at. Evaluating
byte-identical retrieval with the local 35B as both reader and judge instead of gpt-4o-mini doing both moved the hybrid’s LoCoMo score by 6.9 points
(0.7130 against 0.7825); both roles change together, so the swing belongs to the evaluation stack as a whole, not to reader quality alone. Separately,
re-judging identical answers under different judge prompts moved category-level accuracy by 5 to 15 points. Set that against the architecture
([Tab. 8](#table-8)): the hybrid and the flat index differ by 0.3 points, and both sit 3.3 to 3.6 from the hosted graph. **On this benchmark, changing
the stack that evaluates the memory matters more than changing which of those three stores you built.** Only Graphiti OSS sits further away than the
stack does, and by a lot: even the cheapest route to it, Zep Cloud to Graphiti OSS, costs 21.2 points, three times the stack’s swing. The scope
matters, because the claim inverts with the haystack: on LongMemEval-M the same hybrid-versus-flat pair that differs by 0.3 here differs by 15
([Tab. 3](#table-3)), better than twice that swing. The evaluation stack dominates where the stores tie; the store bundle dominates where the
benchmark actually stresses it.

That last gap needs care, because three different things sit behind that number and conflating them makes it unreadable:

**The reader** sits at the end and answers from whatever context it is handed. It is identical for every store, which is what makes the store comparison valid at all; the 6.9-point row swaps it together with the judge.**The extractor** sits at the start, inside the store, and decides what ever gets written down. It is part of the store being compared, not part of the harness around it.**The pipeline** is what the extractor runs in: how many passes it makes over each message, what it resolves, what it keeps.

So which of the three explains the 21 points between the last two rows of [Tab. 7](#table-7), the hosted Zep Cloud at 0.7461 and the self-hosted
Graphiti OSS at 0.5338? Not the reader: it is the same model for both. Not the extraction model’s capability either, because the open engine ran with
the same class of extractor the vendor’s published numbers were built with and still landed where it did. What is left is everything the hosted
service does around that extractor, and this is the point where the comparison reaches the edge of what it can honestly claim.

Zep Cloud is a hosted product, and its row here is Zep’s own published context. I can measure what reaches the reader on each side, not the ingestion
or ranking that produced it. The open engine’s contexts are visibly thinner, carrying fewer stored facts and much shorter entity summaries, but that
is a property of the released artifacts rather than a description of anyone’s internals. One known difference does not favour them: their published
ingestion reads a `blip_captions`

key where the LoCoMo field is `blip_caption`

([zep-papers#9](https://github.com/getzep/zep-papers/issues/9)),
dropping image captions that my runs kept.

None of this suggests their published number is wrong. It reproduces from their own released grades, and their contexts re-scored under this study’s
reader and judge give the 0.7461 in [Tab. 7](#table-7), within a point of their published 75.14. The 21-point gap is real and reproducible; its cause
is not observable from outside. Read “the hosted pipeline” as a label for the part I could not see, not a mechanism I verified. A store can only
answer from what its extractor wrote down, however good the reader in front of it, and that is why importing a number from someone else’s protocol
tells you nothing.

| What changed | From | To | Points lost |
|---|---|---|---|
The reader+judge stack, retrieval byte-identical | `gpt-4o-mini` 0.7825 | `Qwen3.6-35B-A3B-mxfp4` 0.7130 | 6.9 |
| The store, reader unchanged | Hybrid 0.7825 | Place-organized 0.7792 | 0.3 |
| The store, reader unchanged | Hybrid 0.7825 | Zep Cloud 0.7461 | 3.6 |
| The store, reader unchanged | Place-organized 0.7792 | Zep Cloud 0.7461 | 3.3 |
| The embedder inside one store | Graphiti OSS 0.5338 | Graphiti bge-m3 0.5286 | 0.5 |
The store, the cheapest route into the open engine | Zep Cloud 0.7461 | Graphiti OSS 0.5338 | 21.2 |

#### Cost

The graph lineage pays twice ([Tab. 9](#table-9), [Fig. 13](#figure-13)). At read time, distillation was supposed to buy density, but the graph rows
hand the reader six times the context of the raw-turn stores (21.5k chars median against 4.0k and 3.5k) and scores lower with it. Graphiti OSS is
leaner at 7.9k and scores lower still, so density alone is not what the hosted pipeline is buying. At write time, the raw-turn stores embed while the
graphs run an LLM over every message, and that difference compounds brutally with history length. Each message costs the graph a reasoning call, or
several: extract the entities and facts, resolve them against the entities already in the store, then check whether the new fact invalidates an old
one. Measured on this hardware that ran about 14 seconds per message, against a few milliseconds to embed one. LongMemEval-M’s haystacks hold roughly
3.7 million messages, so the arithmetic lands at about 600 days of single-stream GPU time to ingest one run (parallelism divides the wall-clock, not
the bill), against hours for the embedding-only stores, which is why the graph lineage has no long-haystack row at all.

Six hundred single-stream days is a fact about my hardware, not about physics, and it is worth saying so plainly because the hosted vendors do not wait for it: the work parallelizes almost perfectly, so a hundred concurrent workers bring the wall-clock to about six days. What no parallelism touches is the unit economics. Every message costs the graph several LLM calls where the raw-turn stores cost one embedding, and at list prices for a small hosted model that is roughly two orders of magnitude more per message: about $14 of ingest for a single long user history, against about $0.03. Graph memory is affordable; it is just priced like a product decision rather than an implementation detail, and the bill scales with everything your users ever said. Context sizes are medians over the same 1,540 questions; the ingest comparison is wall-clock on identical hardware.

| Store | Median context / question | Ingest |
|---|---|---|
| Hybrid | 4.0k chars | Embedder only: ~$0.03 per long user history |
| Place-organized (MemPalace) | 3.5k chars | Embedder only: ~$0.03 per long user history |
| Entity-and-time (Zep Cloud) | 21.5k chars | LLM per message: ~$14 per long user history |
| Entity-and-time (Graphiti OSS) | 7.9k chars | LLM per message: ~600 single-stream GPU-days, or ~$7,000, per -M run |

## What file-based memory is actually good at

A fair comparison has to state the other side, because file-based memory is popular for real reasons, and none of them are refuted by the numbers above.

**It is human-readable and human-editable.** Your memory is a folder of markdown files. You can open it, read it, fix a wrong fact, delete a stale one, or commit it to git. A vector store is opaque by comparison. For a tool you operate yourself, this is worth a great deal.**It has zero infrastructure.** No embedder, no vector index, no background workers, no graph. If your agent already has file tools, memory is free to add. Structured memory is a small distributed system you have to run.**Its writes are distillation.** Because the model decides what to keep, each memory is a considered lesson, not a raw fragment. At small scale that curation produces a genuinely tidy, high-signal store, which is exactly the regime a personal coding assistant lives in.**Its reads are cheap when the store is small.** Everything above is the large-history regime the benchmark stresses. When the whole memory fits in the index, the read path is just the index in context, and the grep never fires. A few hundred lines of curated notes covers a lot of everyday use.

Read the accuracy numbers with that scope in mind. The benchmark deliberately lives in the hard regime: dozens of sessions, facts scattered across
them, questions that force a join. That is the regime where curation forgets and a small index cannot hold enough, and it is the regime a memory
*product* has to survive. It is not the regime a single-project assistant with fifty lines of notes lives in, and there it is not just adequate, it is
the better engineering trade. The regime-dependence cuts both ways, and it is worth being honest about: on easier, factual-recall benchmarks the gap
narrows sharply. Letta, a memory startup, [reported 74% on LoCoMo](https://www.letta.com/blog/benchmarking-ai-agent-memory/) using nothing fancier
than plain files, and argued a filesystem may be most of what you need. The structured store earns its keep specifically where the questions force
joins across many sessions, which is the part of the problem I find most interesting and the part a memory product cannot dodge.

## Remembering what worked: the agentic benchmarks

Everything above measures one kind of remembering: what was said. An agent accumulates the other kind too, what *worked*: the know-how of past
attempts. The natural question is whether this post’s architectures carry over, so the same structured store was put to work on the agentic benchmarks
the memory-training literature uses: [ALFWorld](https://alfworld.github.io/) (household tasks in a text world: find the mug, heat it, put it away) and
[WebShop](https://webshop-pnlp.github.io/) (find and buy the right product in a catalog, scored with partial credit). Memory here is an **experience
bank**: training-split episodes distilled into atomic entries (a task pattern, the moves that worked), embedded, and retrieved top-k into the acting
model’s prompt. Nothing is trained; it is the same architecture as the conversational study, wearing different content.

The bar in this realm is MemHarness, the experience architecture described earlier: the 7-billion-parameter policy whose retrieval, critique, and reconstruction were trained by reinforcement learning. Being trained rather than bolted on turns out to be the whole story. Two untrained actors ran with and without the experience bank, everything else frozen, and each benchmark gets its own subsection below, mirroring the evaluation section. Two reading notes apply to every table: MemHarness’s out-of-distribution (OOD) number is the comparable one, since the untrained rows run unseen splits; and “no memory” removes only the experience bank, the harness’s task scaffolding (playbooks, target hints, loop guards) stays in every untrained row, so the ablation isolates retrieved memory. There is no cost subsection here, deliberately: the three pay in currencies that do not share an axis (self-hosted GPU time, API dollars, training compute), and charting the two we measured would imply a comparison the third cannot join.

### ALFWorld

[ALFWorld](https://alfworld.github.io/) is a text world of household tasks: find the mug, heat it, put it away. Six task categories, binary success
per game, scored as macro SR (success rate averaged over categories, so no category dominates). The evaluation runs the 134 unseen games.

#### Accuracy

| Actor | Macro SR |
|---|---|
| Local 35B, no memory | 0.603 |
| Local 35B + experience bank | 0.645 |
Frontier actor (`claude-sonnet-5` ), no memory | 0.959 |
| Frontier actor + experience bank | 0.973 |
| MemHarness (GRPO-trained 7B) | 0.852 / 0.859 OOD |

The ablation runs at both actor tiers, and the deltas line up as the headroom pattern predicts: the weak actor gains 4.2 points from the experience bank, the frontier actor 1.4. Only the direction is claimable, though. The weak actor’s gain rescued 16 games and cost 10, which an exact paired test puts at p = 0.164, inconclusive under the pre-set threshold, so ALFWorld corroborates the pattern without carrying it; the significant weak-actor evidence is WebShop’s alone. One detail inside that null is worth keeping: five of six categories move positive with memory, while the category the 35B fails by looping is unmoved to three decimals. Retrieved experience does not repair a policy-level failure mode.

At frontier quality the benchmark saturates instead ([Tab. 10](#table-10), [Fig. 14](#figure-14)): the tasks yield entirely to strong reasoning
(ten-step solves, four of six categories perfect), memory rescued exactly 2 games and hurt none (p = 0.5), and the untrained frontier baseline exceeds
MemHarness’s number. That last fact is not claimed as a beat: it is actor class plus harness scaffolding, not a method comparison. The right reading
is that ALFWorld’s bar is procedural competence, find the object, use the appliance, with a ceiling any sufficiently strong actor reaches, and there
are two routes to that ceiling: MemHarness trained the competence into a 7B; this study rented it from a frontier model. Memory is rounding error on
both routes (ours +1.4 points at p = 0.5, theirs +2.2, and their own ablation shows raw replay *hurts* their trained policy), so lining the results up
gives an ordering that is actor class all the way down: frontier, then trained 7B, then scaffolded 35B, then a frontier model on their plain
scaffold-free harness (their strongest closed-model row, 62.1). Memory has no headroom left to buy here, the agentic twin of the head-to-head’s
“hybrid ties a flat index” finding: the actor dominates, memory works the margin.

### WebShop

[WebShop](https://webshop-pnlp.github.io/) is product search against a 1,000-item catalog with a purchase at the end: 500 test sessions, a 15-step
budget. Two metrics bracket it: **score** grants partial credit for a near-miss purchase, and **SR** counts only perfect ones. The small catalog often
contains no exact match for the instruction, which caps attainable reward for every actor.

#### Accuracy

| Actor | Score | SR |
|---|---|---|
| Local 35B, no memory | 63.5 | 0.376 |
| Local 35B + experience bank | 66.0 | 0.418 |
Frontier actor (`claude-sonnet-5` ), no memory | 65.1 | 0.444 |
| Frontier actor + experience bank | 65.2 | 0.450 |
| MemHarness (GRPO-trained 7B) | 87.4 | 0.756 |

WebShop delivers the agentic benchmarks’ only significant memory-ablation effect, and their clearest boundary ([Tab. 11](#table-11),
[Fig. 15](#figure-15)). The weak actor gains 4.2 points of success rate from the bank (paired McNemar, p = 0.022), exactly where theory puts it: far
from its ceiling, in a domain where episode know-how (query phrasing, option discipline, the scoring rules) transfers between tasks. The frontier
actor gains +0.6 points of success rate (p = 0.8), landing at nearly the same total as the 35B; the trained policy’s 0.756 shows that what binds the
untrained arms is not the catalog alone but what untrained interaction with the hidden rubric extracts. Note what that ceiling does to actor class:
the same actor swap that buys 31 points of success rate on ALFWorld buys +1.6 score here, because WebShop’s reward is shaped by the catalog and its
partial-credit mechanics, not by actor smarts. The deeper difference between the two benchmarks is what each one hides. ALFWorld states its goal in
the observation, so success yields to reasoning, and the one frontier actor tested gained nothing detectable from the bank there (0.959 against 0.973,
p = 0.5). WebShop grades with a rubric the agent never sees, weighing attributes, options, and price into partial credit over a catalog that often has
no exact match, and no amount of reasoning over the observation reveals how that grader will score a near-miss. In these experiments nothing taught it
except training against the reward signal itself, and retrieval never sees the reward: a bank stores what the agent did, not what the grader thought
of it. That is why only training reaches the bar: every training-free arm lands at a score of 63 to 66 against MemHarness’s 87.4, and their number
comes from reinforcement learning against the environment’s own reward, which teaches the policy the reward’s *mechanics*, when to settle for a
partial match, when to stop browsing, what an option is worth. Neither prompting nor a stronger actor replicates that, and none of the stores tested
here closes the gap from the outside.

That claim can be probed from the inside too, with one hard scope limit stated up front: the probe runs on my port of their frozen 7B actor, and that
port falls well short of their published baselines, so it can speak about this port, not about their published system. Hold the port fixed on WebShop
(full catalog, 500 sessions) and swap what its memory holds. Its own released 7,859-episode bank, injected through its own wire format and retrieval
semantics, scores 69.5 with a 0.306 success rate against the no-memory 71.0 and 0.300 (p = 0.69). The same bank under a different retrieval semantics,
situation-match instead of memory-text match, scores 69.1 and 0.298. On strict success the three arms are statistically indistinguishable; on
partial-credit score both memory arms sit nominally *below* the no-memory baseline (by 1.5 and 1.8 points, the latter’s paired interval excluding
zero), so if anything the bank costs this port a little. What that establishes: the tested injections did not help this port, so the bank does not
carry its value in a form that survives being bolted on from outside. What it cannot establish: that bank content or retrieval plays no role in the
published system, or that the training is the cause. The training remains the leading explanation because their own ablation points the same way from
the trained side, with raw replay *hurting* their trained policy, the same null as our frontier arms.

Lay the results beside each other and one pattern organises the realm: **memory paid only where the actor had headroom.** Two actor tiers on two tasks
cannot establish a law, and the weak-versus-frontier difference in memory benefit is itself not significant; but every observation lines up the same
way. A weak actor far from ceiling gains real points from retrieved experience; a frontier actor gains nothing detectable, having already reached what
untrained interaction with the task seems to extract (the trained policy’s higher bar shows the task itself is not saturated); a policy trained for
the task is actively hurt by raw replay. This is the consolidation null from earlier in the post seen from the other side: there a capable reader
absorbed the store’s disorder and left tidying nothing to buy, here a capable actor absorbs the task and leaves memory nothing to buy. Training buys
its bar at the price of narrowness, too: served frozen outside its own harness, the released MemHarness model is acutely sensitive to exact prompt
format, the specialization reinforcement learning produces, where a frontier actor’s robustness is precisely the thing you rent. That places every
retrieval-shaped system in this post, files, stores, temporal graphs, and hybrids alike, on one side of a line: bolt-on memory, model-agnostic, paid
for at write and read time, its value floating on the gap between the actor and the task. MemHarness sits on the other side: memory as trained
behavior, paid for in training compute, inseparable from its actor. The conversational benchmarks reward the first kind everywhere; the agentic
benchmarks reward it only while the actor is weak; past that line the question stops being “which store” and becomes “whose weights”.

One more disclosure belongs with the bar itself, because it cuts against the comparison. MemHarness’s numbers above are quoted from its paper. Running
their *released* model on my own serving stack, under a faithful port of their harness and prompts, does not reach them: 0.581 macro on ALFWorld
against their 0.830 without memory, and 71.0 score with a 0.300 success rate on WebShop against their 87.4 and 0.756. The shortfall has the same
signature on both benchmarks: the approach reproduces and the precision does not, exact option matches on WebShop and multi-step thermal sequences on
ALFWorld, a shape consistent with serving numerics and 8-bit quantization of a sharply peaked policy rather than with anything about memory, though
that attribution is untested (the separating run, bf16 against 8-bit, was not made). I report it because it makes the comparison’s frame explicit.
Their published bar stands as published; my arms are measured on my stack; and the distance between those two statements is the same cross-stack
caution this post applies to every other number it does not own.

Provenance, disclosed: WebShop’s official dataset is org-locked, so the runs used the community mirror of the same files (1,000-product setting); all frozen arms are single runs, and the memory ablations are paired per-episode. The frontier arms cost about $22 of API spend on ALFWorld and $34 on WebShop, the latter including two voided protocol iterations.

## Takeaways

**The store architectures are bets about where memory’s cost sits, and each is right somewhere.** File-based memory bets on the model’s judgment and a filesystem, pays in model tokens at write time and literal search at read time, and buys transparency and simplicity: the right bet when memory is small, human-owned, and secondary to the task. Structured memory bets on an embedder and a ranker, pays in infrastructure, and is built for recall that holds up as history grows; the long-haystack pair here is consistent with that bet, though no scaling curve was measured.**When memory is the task, structure wins on both axes at once.** Under a fixed model on a hard benchmark, the structured store beat files by**28.7 points on held-out questions**(95% CI [22.1, 35.4]) at a fraction of the measured model tokens per correct answer.** Sparse memory abstains for free.**File-based memory wins the questions whose right answer is “I don’t know”, on both benchmarks: remembering less means over-answering less. Build the structured kind and you must budget for an abstention discipline.**Raw dated facts beat LLM-distilled graphs, and cost less twice over.** Inside the structured family, a good ranker over raw facts beat the graph lineage on the benchmark the graph is sold on, while the hosted graph spent six times the reader context to score lower. Whether place-plus-time beats a flat ranked query depends on where you ask: no detectable advantage on LoCoMo, a significant one on LongMemEval-M’s long haystacks, and no scaling curve connecting the two, which differ in more than history length.**Reasoning at ingest is a product decision, not an implementation detail.** A graph store spends several model calls on every message a user ever sends, where a raw-turn store spends one embedding: roughly two orders of magnitude more, about $14 to ingest one long history against about $0.03. It parallelizes, so it is a bill rather than a wall, but the bill scales with everything your users ever said. It is also why this study has no graph row on the long-haystack benchmark: that one row would have cost about 600 single-stream GPU-days of compute, or about $7,000 of hosted inference, to fill.**Consolidation is the third path, and the measured result is a null.** Both store architectures can run a background pass that reorganizes what is already stored, promoting what gets used or merging what repeats. Built here, it merged real duplicates and bought no accuracy at the scale tested. The design logic says it should pay once the mess outgrows what the reader can hold; that remains a hypothesis, since no consolidation-positive regime was measured.**The ruler can outweigh the architecture, and which one dominates depends on the benchmark.** Swapping the reader-and-judge stack moved a score by 6.9 points on byte-identical retrieval, more than the gaps between the three stores that work where they tie (0.3 to 3.6 points); on the long haystack the same store pair separates by 15. No single benchmark ranks these systems, and no number means anything without its protocol attached.**Retrieved memory paid only where the actor had headroom.** The agentic benchmarks draw the sharpest boundary in this post, as an observed pattern on two actor tiers and two tasks rather than a law: real points under a weak actor, no detectable effect under a frontier one, harm under a policy trained for the task. Where the task yields to reasoning, a frontier actor reaches the trained bar with no memory at all; where the reward has a structure only practice teaches, training stands alone, and it buys that bar at the price of narrowness. Below that line the architectures in this post are the game; at the line, the game becomes training.

Both headline numbers came from the same model answering the same questions. The only thing that changed was the shape of what it remembered with.

## Appendix: methods and caveats

*The story is complete without this section. What follows is the fine print: how the questions were split, every limitation I know about, and the
caveats each number carries. Nothing here overturns a result, but it tells you how far each one can be trusted.*

### How the questions were split, and what “tuning” means

The 500 LongMemEval questions were split once, before any runs: a seeded, stratified 144-question tuning set, a 256-question holdout, and 100 left in reserve. Tuning-set numbers are provisional and never the headline; the holdout is scored exactly once per configuration. “Tuning” means prompts and configuration only, the model’s weights never move: the shared answering discipline (a six-round tool budget, an output cap, a truncation-rescue step, a facts-then-dates-then-verify format), the structured arm’s retrieval depth (top-12) and question-blind recency preload, and the file arm’s index budget (200 lines or 25 KB) and recall tools. The file-based arm got an equal tuning budget, and two of its frozen settings came from watching its own failures: the same truncation rescue the other arm has, and a guard against index-wiping writes. The holdout then judged both arms with everything frozen.

### Limitations

**The shared answering prompt has a heritage.** Its discipline was developed in earlier work whose read path resembled the structured arm’s. Equal tuning budget in this study is not equal ancestry; a file-native prompt built from scratch might serve that arm better, and I did not build one.**The file-based arm is a reconstruction, deliberately.** Its curation instructions are a faithful-as-documented rewrite of an unpublished original, every mechanism decision traced to a cited public source in the released spec. I chose not to validate against the shipping CLI: a closed-product run is a snapshot of whichever model version shipped that week, unreproducible by design. The fidelity burden is met by the traceable spec, the oracle control below, and a published envelope check (save rates, index shapes, read patterns) that anyone with the real tool can run to falsify the reconstruction.**One model family judges itself, and an independent judge has now audited part of it.** In the main experiment the judge is identical across arms and finished with a deterministic refusal pass, but it shares a model family with the arms it scores. For the store head-to-head, that concern has been tested: a frontier judge from a different vendor re-scored every arm’s published responses on a frozen 100-row sample. It grades uniformly stricter (4 to 7 points on every arm, agreement 0.91 to 0.96, Cohen’s kappa 0.82 to 0.89, with a 0.05 agreement spread that bounds agreement but cannot rule out directionally different errors per arm), and every ranking survives, with the hybrid’s win over the graph vendor significant on a tenth of the data (p = 0.024). The supported claim is rankings preserved on that frozen sample. The main experiment’s own judge remains unaudited.**The holdout was widened once, and the verdict stayed on the blind set.** Mid-study I extended the held-out set from 256 to all 356 non-tuning questions; the added questions had never been run or tuned on, but one arm’s solo holdout score had been seen, which is partial peeking. An independent review called it, so the verdict was locked on the blind 256. The tables in this post report all 356, because that is the set whose per-question rows are published, and widening changed the gap by 0.0002, so the two agree to the fourth decimal. The commit history proves the ordering.**Truncation hit the arms unequally**(20 of 144 file-based answers against 3, under a busy serving layer). The frozen rule counts both as wrong symmetrically, and the gap survives excluding every truncated row.**Absolute numbers reflect a minimal harness.** Both arms run a deliberately minimal shared loop because mechanism isolation is the point. A richer loop can help or hurt; for calibration, one production configuration of the structured mechanism reaches the mid-0.80s on this benchmark’s held-out questions.**The oracle control splits the gap descriptively, not causally.** Answering with the file arm’s*entire*memory directory in context and no tools scores 57.1% held-out, between the file arm’s 44.9% and the structured arm’s 73.6%: in aggregate, 12.2 points of the gap sit on the read side (saved but not found) and 16.5 on the write side (never written down). Per question the interventions are non-monotonic (the oracle also changes how the reader sees the store, and it loses questions the file arm won), so this is an aggregate split between arms, not an identification of where each point was lost. On preference questions the oracle beats the structured arm (72.2% against 61.1%), consistent with a search miss; on abstention it scores exactly what the file arm scores, which suggests over-answering tracks eager retrieval rather than context volume.**The comparison is measured at one history length.** Both arms ran at roughly 47 sessions per question. The 500-session variant is another two to three days of runs at ten times the history, outside this study’s window, so the prediction that files fall further behind as histories grow ([Tab. 3](#table-3)) is untested rather than confirmed. The long-haystack rows that do exist compare two structured stores, not files against structure.**Wall-clock is indicative, not controlled**; token counts are the load-independent cost metric.** Reproducibility has a scope.**The baseline, file arm, judge, and analysis run against any OpenAI-compatible endpoint; the structured arm calls a memory service, and its raw per-question rows are published for inspection either way in[a40-labs/memory](https://github.com/a40-labs/memory). The protocol, including every amendment and its timing, was recorded in the study’s private repository before the scored runs; the public repository carries the artifacts and an explicit ledger of what it cannot substantiate, not a timestamped registration.

### Reading LoCoMo

**Retrieval recall is not answer accuracy**, and LoCoMo numbers in the wild mix the two: a store can be scored on whether the right item merely
surfaces in its top-k, or the whole system on whether it answers correctly end-to-end. The gap is structural; this study’s store surfaces the right
session about 95% of the time on LongMemEval while its end-to-end accuracy sits at 0.73, the answering step consuming the rest.
[MemPalace](https://github.com/mempalace/mempalace) publishes *only* retrieval recall on LoCoMo, with an explicit no-QA disclaimer, while the QA
numbers people quote (Zep’s disputed 84 / 75.14 / 58.44, mem0’s ~67) are LLM-judged answer accuracy from entirely different systems; putting one next
to the other is comparing different sports. Beyond that, the benchmark is contested ground with no agreed state of the art and no clean third-party
reproduction of the leaders, and the research community has argued it is close to saturated. Its questions cluster inside 10 conversations, so honest
confidence intervals must resample conversations, not questions. And its adversarial category inverts the incentive, rewarding refusal and punishing
exactly the eager retrieval that helps everywhere else, which is why every LoCoMo score in this study is published under both scopes. One disclosure:
the file-based arm’s LoCoMo run predates a serving-layer retry fix, and 18 of its 300 answers died to output truncation and count as wrong; its
numbers are floors.

### Head-to-head boundaries

Every head-to-head row is a single run, and identical reruns at temperature 0 drifted by 0.3 to 0.4 points, so the last decimal is noise. Graphiti-OSS
is a best-effort parity configuration of the vendor’s open-source engine, not their hosted product. The hybrid’s retrieval was verified against a call
ledger to confirm its reranker actually ran on every row, because that component fails open (a degraded run returns a full, silently unranked result
set; 295 early rows failed exactly that check and were quarantined and re-run). What is published, stated precisely: per-question verdicts, hit
counts, and context sizes for every head-to-head arm, with the shared judge prompt, in [a40-labs/memory](https://github.com/a40-labs/memory), where a
verifier re-tallies every score and re-derives every statistic. The retrieved contexts, answers, and grades themselves are not republished: Zep
Cloud’s contexts are their data ([their repository](https://github.com/getzep/zep-papers) has them), and the full bundles for the other arms live in
the study archive. The published artifacts support re-tallying and re-deriving, not independent re-judging.

### A side note for Obsidian users

Many people wire an [Obsidian](https://obsidian.md/) vault (markdown notes joined by `[[wikilinks]]`

) into an agent as its memory, usually over MCP;
[Basic Memory](https://github.com/basicmachines-co/basic-memory) is the clearest example. By this post’s taxonomy that is the **file-based** kind:
storage the model curates, read back by text search. What the vault adds is an explicit link graph, but authored at write time by hand or by the model
(one more thing curation has to get right) and walked deterministically rather than ranked: a graph doing a retriever’s job through foresight. The
vault crosses toward the structured kind only when an embedding index is bolted on
([Smart Connections](https://github.com/brianpetro/obsidian-smart-connections), Obsidian Copilot), which swaps the grep for exactly the ranked read
path the structured store uses while the storage stays plain markdown. The honest picture is a spectrum: a plain vault sits with the file arm, an
embedded vault sits closer to the structured one, and the write-side curation cost this post measured is paid the whole way across, right up until you
stop asking a model to decide what to keep.