40 tasks. Without memory it got them wrong 52.5% of the time. With memory: 0%.
Not "fewer mistakes". Not "improved by 30%". Zero. Forty out of forty.
Here is the entire result before I explain anything:
| Claude Code | Claude Code + RE-call | | |---|---|---| | Got it wrong | 52.5% of runs | 0% | | Answer correctness | 0.144 | 0.358 | | Factual correctness | 0.288 | 0.559 | | Times memory made things worse | 0 out of 40 |
Correctness doubled. Factual accuracy doubled. And across 40 head to head runs, the version with memory never once lost a round the other one won.
The p values are 0.0003 and below. In plain language: this is not luck.
Numbers are nice. Here is what they mean on a Tuesday afternoon.
Your agent stops confidently recommending commands that do not work. Not commands that error out, because you would catch those. Commands that succeed, print something reassuring, and do nothing at all. Those are the expensive ones, precisely because you believe them.
Your agent knows the decisions your team already made. Why that library version is pinned. Why that directory is gitignored. Which flag broke production in March. The knowledge that lives in people's heads, old pull requests, and nowhere the model can see.
Answers about your repository get twice as correct. Not longer, not vaguer. More right.
It never makes anything worse. Zero regressions across 40 paired runs. That is the number I would want to see before letting anything near my workflow.
I have run RE-call through plenty of them. Retrieval benchmarks, long context benchmarks, answer quality benchmarks. They are genuinely useful and I keep running them, because a benchmark tells you where your system is weak, which knob actually matters, and whether last week's idea was clever or merely expensive.
But a retrieval benchmark proves your retriever retrieves. It does not prove anyone is better off.
You can top a leaderboard and change nothing about a single developer's day. That gap bothered me enough to go and measure the day instead.
The whole design fits in one sentence: run the same task through two Claude Code sessions that differ by exactly one thing, and count what happens.
The two sessions. Both ran the same model. Both got the same tasks. Both loaded the same CLAUDE.md
project file, byte for byte, and I made the harness verify that at the start of every run rather than take my word for it. Both ran with hooks, plugins and auto loaded context switched off, so nothing could sneak in a hidden advantage. The only difference: one session could also search a curated memory of the project, and the other could not.
The tasks. Ten tasks, each hiding a hazard this project ran into once, paid for, and wrote down. A CPU cap that reports success and caps nothing. An image renderer that does not exist on this machine. A library install that quietly breaks the environment. Ordinary requests, each with a booby trap that only bites if you do not know the history.
The scoring for hazards. Each hazard has a deterministic checker that reads the session transcript and decides whether the agent walked into it. A pattern, not an opinion. No model gets a vote on this part.
The repetitions. Each hazard task ran ten times per arm, because agents are not deterministic and one run of anything proves nothing. Sixty paired runs in total.
The honesty gate. Before any pair counts, the harness checks that the memory tools were actually present in the memory session and actually absent in the other one. If either is wrong, the pair is thrown away rather than scored.
The predictions. I wrote down what I expected before running anything, and committed it with a timestamp. That is the only reason I can tell you honestly which results surprised me, rather than building a story around whatever came out.
The statistics. Every comparison is paired, same task through both arms, so the test is on the difference within each pair rather than two independent piles of numbers. Hazards get an exact McNemar test, which looks only at pairs where the two arms disagreed. Costs get a Wilcoxon signed rank test, which does not assume the data is nicely shaped, because agent token counts absolutely are not.
Counting hazards is the easy half. "Was the answer actually any good" is where most demos quietly stop and switch to screenshots.
For that I used ** Ragas**, an open source evaluation toolkit for LLM applications. It exists to turn "that looks about right" into a number, which is exactly the problem I had. The way it works is refreshingly boring. You write down the correct answer in advance. Ragas then breaks the agent's actual response into individual claims and checks each one against your reference, so you find out how much of what it said was right, and how much was confidently wrong.
Two metrics from their library did the work:
AnswerCorrectness
weights=[1.0, 0.0]
, which turns off the embedding similarity half. That half rewards an answer for FactualCorrectness
Three things kept it honest:
Final tally: 120 responses scored, zero failures. This is the part product demos skip, and the part I would look for first as a reader.
I also ran tasks where the answer was already sitting in CLAUDE.md
. Both agents had that file. So memory should add precisely nothing.
Result: 0.083 against 0.083. A difference of exactly zero. p = 1.0.
If memory had "won" there too, my experiment would be broken and this article would not exist. The gain appears exactly where memory holds something the file does not, and vanishes everywhere else. That is what a real effect looks like, as opposed to a flattering one.
The first time I ran this, I did the clever thing. I gave the agent RE call instead of CLAUDE.md
. Retrieval replaces your documentation. That is the entire pitch of RAG.Answer quality improved by 0.044, with a p value of 0.43. Translation: nothing happened.Then I ran it the way an actual developer would set it up. Keep your CLAUDE.md
. Add memory on top.
Quality nearly doubled.
Replacing your documentation with retrieval does nothing. Adding retrieval to it changes everything.
I had written down the opposite prediction before running a single session, which is the only reason I can prove I was surprised rather than inventing the narrative afterwards.
About 15,000 extra input tokens per session, and a couple of seconds. On tasks where memory cannot possibly help, it runs slightly slower, because it goes and looks something up and finds nothing useful.
That is the trade: pay a little on every task, stop paying enormously on the ones that would have gone wrong. Roughly once in twenty runs, the agent without memory disappeared down a hole for seven minutes chasing something that had been written down all along.
I would rather say this myself than have someone say it for me.
This shows memory eliminates a specific, expensive class of mistake and makes answers about your project substantially more correct. It does not show memory makes an agent generally smarter. Where memory has nothing to add, it adds nothing, and I have the control numbers proving both halves.
It also does not yet show that the work succeeds more often. Every task here asks for a recommendation and scores whether that recommendation is right. None of them writes code, runs a test suite, or ships anything.
The next round is already running, and it asks the harder question: does the work actually succeed?
Real tasks, in a real repository, where a test suite decides the outcome. No judge, no reference answer, no interpretation. It passes or it does not. Alongside it, more measurements on where the gains come from: how often memory gets consulted, how often it returns the fact that matters, and which kinds of task benefit most.
That is the number I actually want, and I will publish it whichever way it falls.
Part 2 coming soon.