# I Gave Claude Code a Brain. Then I Measured What It Costs to Ask for One. > Source: > Published: 2026-09-01 15:57:06+00:00 At the end of [part 2](https://dev.to/gde03/i-gave-claude-code-a-brain-this-time-i-let-the-tests-judge-it-5h0p) I promised a clean competitor comparison: more than one memory system, the same admission gate, enough capacity to finish. This is that run. 1,453 agent sessions, four arms, five corpus conditions, every outcome graded by executing a checker against the code the agent actually wrote. No LLM judge anywhere. Each condition is a **4,911 document corpus, about 27,000 embedded chunks**. Of those documents, **4,902 are adversarial distractors** generated from the corpus's own vocabulary to be confusable, and the rest are the planted material for 36 tasks. That ratio is the point. On the small feed I used earlier, retrieval hit@10 was 1.000: everything was findable, so the benchmark could not tell "the product retrieved badly" from "the agent never searched". With the haystack, BM25 hit@1 drops from 0.485 to 0.182. The five conditions differ in what is planted, and **four of the five are traps**: | condition | what the corpus holds | share of cells | |---|---|---| `present` | the governing fact, cleanly | 37% | `superseded` | the fact, beside an outdated version of itself | 15% | `adjacent` | something related and inapplicable | 16% | `absent` | the fact deleted, distractors left behind | 17% | `contradictory` | two rival versions, no authority between them | 15% | **Only 37 percent of cells contain a clean fact worth retrieving. In the other 63 percent the corpus is outdated, contradictory, inapplicable, or empty**, and a memory layer's job there is to not make things worse. I built it that way deliberately. A benchmark where memory can only help is a demo. Two commenters asked for a placebo. Fair: if you tell an agent "you have memory, consult it before acting" and it then does better, you have not separated the memory from the telling. So I built an arm carrying **the complete memory instruction and no memory at all**. Same protocol text, byte for byte, as the two real arms. Nothing behind it. | comparison | wins | losses | net | p | |---|---|---|---|---| | instruction only, vs bare | 26 | 43 | -17 | 0.053 | **Telling an agent to consult memory, when there is no memory, makes it measurably worse.** Seventeen cells of 358, borderline significance, in the direction nobody advertises. That reframes everything. "Does memory beat no memory" is not directly answerable, because every memory product requires that instruction. The answerable question is: **does the product earn back the cost of asking for it?** [RE-call](https://github.com/GiulioDER/RE-call) `0.11.0` is mine, so read accordingly. [MemPalace](https://pypi.org/project/mempalace/) `3.8.0` is the better known of the two. Both received the same instruction and the same gate. Every write tool was withheld from every arm. | comparison | wins | losses | net | p | |---|---|---|---|---| RE-call vs instruction only | 41 | 21 | +20 | 0.015 | | MemPalace vs instruction only | 25 | 23 | +2 | 0.885 | | RE-call vs MemPalace | 48 | 30 | +18 | 0.054 | | RE-call vs bare | 47 | 44 | +3 | 0.834 | | MemPalace vs bare | 38 | 53 | -15 | 0.142 | McNemar exact on discordant pairs. **The only comparison clearing p < 0.05 is RE-call against the instruction-only control.** Note the fourth row before anyone else does. **Against a bare agent, RE-call is +3 cells at p = 0.834, which is nothing.** The win is real against the instruction the product requires and invisible against an agent you never asked. A sceptic will quote that row and is entitled to. Tasks solved out of cells, admitted only where all four arms produced a record. Ordered by how much RE-call gains over the instruction-only arm, which is the comparison that isolates the product from the coaching. | condition | cells | bare | instruction | RE-call | MemPalace | RE-call gain | |---|---|---|---|---|---|---| present | 134 | 52/134 · 0.388 | 53/134 · 0.396 | 67/134 · 0.500 | 56/134 · 0.418 | +14 | superseded | 52 | 38/52 · 0.731 | 34/52 · 0.654 | 41/52 · 0.788 | 35/52 · 0.673 | +7 | | absent | 60 | 49/60 · 0.817 | 41/60 · 0.683 | 44/60 · 0.733 | 37/60 · 0.617 | +3 | | contradictory | 54 | 44/54 · 0.815 | 41/54 · 0.759 | 39/54 · 0.722 | 42/54 · 0.778 | -2 | | adjacent | 58 | 42/58 · 0.724 | 39/58 · 0.672 | 37/58 · 0.638 | 40/58 · 0.690 | -2 | all | 358 | 225/358 · 0.628 | 208/358 · 0.581 | 228/358 · 0.637 | 210/358 · 0.587 | +20 | **RE-call is the only arm that beats doing nothing, by 0.9 points overall.** MemPalace at 210/358 sits below the no-memory baseline and barely above instruction-only. MemPalace's gains against the same control are +3, +1, -4, +1, +1, totalling **+2**. The shape of that table matters more than the total. **Memory pays where there is something to recall and costs where there is not.** On `present` RE-call converts 67 of 134 against bare's 52. On the three conditions built to punish retrieval it is flat or slightly negative, because surfacing something inapplicable is worse than surfacing nothing. Go back to the composition. **Sixty-three percent of these cells are conditions built to punish a memory layer**, where the corpus is outdated, contradictory, inapplicable, or empty and the best available outcome is to waste nothing. Only 37 percent contain a clean fact worth retrieving. **RE-call is still the only arm that finishes above the no-memory baseline.** It does that while spending nearly two thirds of its cells in terrain designed to make retrieval a liability, and while paying the -17 cell entry fee that the instruction alone costs. That is the number I would defend hardest, because it is the unflattering way to measure my own product and it still comes out positive. **And the composition is a choice I made, not a fact about anyone's repository.** Real memory does not sit still at that ratio. How much of it is current, governing and correctly superseded is a property of how the layer writes and maintains notes over time, which **this benchmark does not measure at all**. It fixes the corpus and varies the product. So read the table as a floor rather than a forecast. On the condition a well maintained corpus is mostly made of, RE-call is +14 of its +20. What that becomes at a healthier ratio I have not measured, and I am not going to put a number on it. I would rather someone else did. A composition sweep, holding the product constant and varying the mix, is the obvious next experiment and it is not run. For each product, the solved rate when it searched minus when it did not: | condition | RE-call | MemPalace | |---|---|---| | present | +0.181 | +0.019 | | adjacent | +0.147 | -0.032 | | contradictory | +0.079 | +0.085 | | absent | +0.031 | +0.264 | | superseded | +0.006 | -0.094 | **When RE-call searches it helps in five conditions of five. When MemPalace searches it hurts in two**, while searching more often (0.58 to 0.69 against 0.46 to 0.56). On `present` , the condition built so the answer *is* in memory, searching is worth **+0.181 to RE-call and +0.019 to MemPalace**. That is the cell both products exist for. Cost, median beside mean because these distributions are skewed enough that on an earlier run of mine the two disagreed in sign: RE-call 65.9 s and 56,476 input tokens per session, MemPalace 68.6 s and 90,706, bare 44.7 s and 14,085. **RE-call is faster and uses 38 percent fewer tokens than MemPalace while delivering ten times the net benefit.** Both are materially heavier than no memory, which is the honest cost of the idea. The previous run said something different, and I want to be the one who says why. In it, RE-call looked like **the most damaging arm of seven**, worst on three of four adversarial conditions. I had written that up as a genuine weakness of my own product. It was my harness. That run gave RE-call **1,958 bytes of coaching over the shared protocol against MemPalace's 853**, and most of the surplus was not about RE-call at all: search before your first write, search by symptom rather than goal. Generic advice that helps any retrieval system, handed to one of them. Worse, the flag selecting the fair variant was **hardcoded in my launcher with no override**, so no run I had ever done could have been matched, and I had not noticed. Under the corrected instruction the finding evaporates. RE-call's damage-only net harm falls from +0.073 to +0.0122, **identical to MemPalace's**, and its `superseded` damage from 0.179 to 0.036. Its usefulness score also fell, 0.377 to 0.281, while MemPalace's did not move at all, because MemPalace had been on the shared protocol all along. My product's advantage narrowed from 1.8x to 1.35x once I stopped giving it a bigger prompt. Both corrections point the same way. I had been generous to myself and unfair to the competitor, and it produced one flattering finding and one damning one, both wrong. RE-call has a reasoning-graph layer. Across 2,181 sessions of the earlier run its four graph tools were called **zero times**. Read alone, damning. It builds that graph from **authored cross-references**, the way a wiki links notes. Its own memory store has 44,364 of them. This benchmark's corpus is rendered agent transcripts: conversation turns, no cross-references. The graph gets entities and no edges. It is a node set, not a graph. **So graph reasoning needs an authored, cross-referenced corpus and this benchmark deliberately measures raw transcripts.** Different corpus classes, not a weakness. Manufacturing cross-references in to make the feature look good would be assuming the answer. I established that with ten read-only queries rather than the 220 sessions the experiment would have cost, after a reviewer pointed out my draft design would have measured a language model reading trusted evidence and published it as a graph result. Two systems is a comparison, not a field, and one of them is mine. The harness is public: [github.com/GiulioDER/agent-memory-bench](https://github.com/GiulioDER/agent-memory-bench). **If you build a memory layer for coding agents, I would rather measure yours than have you take my word for this.** Every adapter is hash-pinned and every vendor is invited to review their own config before a measured run, which is what `adapters//VENDOR_REVIEW.md` is for. Open an issue and I will help wire it up. **And if you just want to check my arithmetic**, the README has a section on verifying a published run with no credentials and no money. The preregistrations are in the repo, including the ones whose predictions I got wrong, and the raw records are published with each run. If you take one number from this, take that one: **63 percent of the cells were built to make memory a liability, and the memory layer still finished ahead of not having one.** One thing I would most like broken: the 45 percent. On `superseded` , the largest single loss is that in 45 percent of sessions **neither** planted document is retrieved at all. That is worth roughly three times more than anything I have measured on the ranking side, it is a query-formulation problem rather than a retrieval-quality one, and it is already preregistered. If someone has a better idea than mine, I would like to lose that one.