I Gave Claude Code a Brain. This Time I Let the Tests Judge It.

A developer's benchmark of Claude Code with a memory layer called RE-call found it improved task success from 36.1% to 58.3% over a static CLAUDE.md file, a 22.2 percentage point gain with a p-value of 0.000145. The test used deterministic checkers across 72 paired comparisons, and RE-call showed the strongest effect on memory-sensitive tasks, improving success by 45.8 percentage points.

The first experiment measured whether Claude Code knew the right answer. This one measured whether the work actually succeeded. In my previous experiment https://dev.to/gde03/i-gave-claude-code-a-brain-its-mistake-rate-went-to-zero-3d8l , I tested whether a memory layer could stop Claude Code from confidently repeating project-specific mistakes. It did. But there was an obvious limitation. The tasks measured whether the answer was correct. They did not ask the agent to modify a repository, run the result, and pass a deterministic test. So I ran the harder version. This time, the agent had to do real work in a temporary repository. A checker decided whether the result was correct. No judge model. No subjective scoring. The task either passed or it did not. Here is the result from the complete run: | Configuration | Successful runs | |---|---| | Claude Code without memory | 50.0% | Claude Code with CLAUDE.md | 36.1% | | Claude Code with RE-call | 58.3% | RE-call improved success over the CLAUDE.md baseline by 22.2 percentage points. The 95% confidence interval was +11.1 to +33.3 points. The paired McNemar test gave a p-value of 0.000145. This was not one lucky demonstration. It was 72 paired comparisons across 24 tasks and three seeds, with zero discarded cells. Each task asked Claude Code to create or modify files in a repository and run the result. The benchmark included tasks involving things like: The important detail is that every task had a deterministic checker. The agent could say "done" all it wanted. The checker only cared whether the repository ended up in the correct state. I ran every task through three configurations: CLAUDE.md The harness also verified that the memory tools were actually available before counting a session. If the memory layer was missing, the entire paired cell was discarded. That happened zero times in the DeepSeek run. The most useful comparison is RE-call versus the static CLAUDE.md file. The static file produced a 36.1% success rate. RE-call produced 58.3%. That is a 22.2 point improvement. The paired results were even more revealing: CLAUDE.md lost CLAUDE.md won only 1 comparison that RE-call lostThe effect was strongest on the tasks classified as memory-sensitive. Across those eight tasks, RE-call improved success by 45.8 percentage points. That is the pattern I was looking for. Memory did not make every task easier. It helped when the task depended on a project decision, convention, or failure mode that was not fully present in the immediate prompt. CLAUDE.md The static CLAUDE.md configuration performed worse than the bare configuration: | Configuration | Successful runs | |---|---| | Bare | 50.0% | CLAUDE.md | 36.1% | The bare configuration won by 13.9 percentage points. This is not what I expected. It suggests that a static instruction file is not automatically a memory system. It can add useful guidance, but it can also add noise, constraints, or instructions that compete with the task. RE-call is doing something different. Instead of putting every historical fact into the permanent prompt, it gives the agent a way to retrieve relevant project history when the current task needs it. That distinction matters. The benchmark also measured what happened inside the memory mechanism. In the DeepSeek run: So the result is not simply that "the model had more tokens." The memory layer was being consulted, and in most cases where it was consulted, it reached something useful. There is still room to improve retrieval. A search is not automatically a good search. The model needs to find the right historical decision, not merely retrieve something related to the words in the prompt. The complete DeepSeek run cost an estimated $0.4964 at the captured API prices. The three arms cost: | Configuration | Estimated cost | |---|---| | Bare | $0.0824 | CLAUDE.md | $0.0863 | | RE-call | $0.3277 | RE-call used about four times as many total tokens as the static prompt configuration. That is the tradeoff. The memory layer costs more on every task, including tasks where memory is unnecessary. In exchange, it can prevent the much more expensive failure mode where an agent confidently follows an outdated convention, repeats a known mistake, or spends several minutes debugging a problem the team already solved. The next engineering question is not whether memory helps. It is how to make the retrieval smaller, faster, and more selective. I also started the same model-freeze run with GPT-5.3 Codex. That run is not a negative result for GPT. The provider hit a credit and in-flight request limit while the benchmark was running. The harness correctly discarded the affected paired cells instead of treating missing sessions as failed model attempts. Only 40 of the 72 paired cells were admissible, so I am not using that run to claim that GPT-5.3 Codex is better or worse. The descriptive numbers pointed in the same direction: | Configuration | Successful runs | |---|---| | Bare | 47.5% | CLAUDE.md | 30.0% | | RE-call | 50.0% | But the run needs to be repeated with the provider capacity issue fixed before it becomes a valid model comparison. That is an operational failure, not a model conclusion. This experiment does not show that memory makes an agent generally smarter. It shows something narrower and more useful: A production memory layer can improve the probability that an agent completes a real repository task correctly. In this run, RE-call improved success by 22.2 percentage points over the static project file. It also improved success over the bare configuration, although that gain was smaller at 8.3 points. That difference is important. The main value is not simply adding more instructions. The value is retrieving the right project-specific context when it matters. The first experiment showed that memory can make answers more correct. This experiment showed that the improvement survives contact with the repository. That is the result I was actually waiting for. The next step is a clean competitor comparison: The goal is not to produce another flattering demo. The goal is to find out whether a memory layer remains useful when the models, vendors, and retrieval systems change. That is the benchmark I want to run next. Memory that abstains instead of guessing. RE-call is agent memory on your own PostgreSQL with pgvector: every hit carries a verdict confidence and provenance, a retracted claim comes back marked superseded , and a question the corpus cannot answer is refused rather than answered from the nearest neighbour ATM-Bench Recall@10 92.8924 against 79.09 for the best published row limits https://github.com/GiulioDER/RE-call/blob/master/docs/ATM BENCH.md · second of ten on MTRAG correct refusals limits https://github.com/GiulioDER/RE-call/blob/master/docs/MTRAG BENCHMARK.md · zero memory-layer LLM calls to build memory, where Mem0 pays one per session limits https://github.com/GiulioDER/RE-call/blob/master/benchmarks/REVIEW.md Why RE-call https://github.com/GiulioDER/RE-call why-re-call · Quickstart https://github.com/GiulioDER/RE-call quickstart · How it works https://github.com/GiulioDER/RE-call how-it-works · Product surface https://github.com/GiulioDER/RE-call product-surface · Documentation https://github.com/GiulioDER/RE-call documentation · Evidence https://github.com/GiulioDER/RE-call evidence Setup guide: install, configure and run RE-call https://giulioder.github.io/RE-call/ · Validity Frontmatter: the open spec RE-call implements https://github.com/GiulioDER/validity-frontmatter Nearest-match retrieval cannot tell the difference between what is true and what merely reads like it. When a corpus keeps its history, and real agent memory does, the retracted…