{"slug": "i-gave-claude-code-a-brain-this-time-i-let-the-tests-judge-it", "title": "I Gave Claude Code a Brain. This Time I Let the Tests Judge It.", "summary": "A developer's benchmark of Claude Code with a memory layer called RE-call found it improved task success from 36.1% to 58.3% over a static CLAUDE.md file, a 22.2 percentage point gain with a p-value of 0.000145. The test used deterministic checkers across 72 paired comparisons, and RE-call showed the strongest effect on memory-sensitive tasks, improving success by 45.8 percentage points.", "body_md": "*The first experiment measured whether Claude Code knew the right answer. This one measured whether the work actually succeeded.*\n\nIn my [previous experiment](https://dev.to/gde03/i-gave-claude-code-a-brain-its-mistake-rate-went-to-zero-3d8l), I tested whether a memory layer could stop Claude Code from confidently repeating project-specific mistakes.\n\nIt did.\n\nBut there was an obvious limitation.\n\nThe tasks measured whether the answer was correct. They did not ask the agent to modify a repository, run the result, and pass a deterministic test.\n\nSo I ran the harder version.\n\nThis time, the agent had to do real work in a temporary repository. A checker decided whether the result was correct. No judge model. No subjective scoring. The task either passed or it did not.\n\nHere is the result from the complete run:\n\n| Configuration | Successful runs |\n|---|---|\n| Claude Code without memory | 50.0% |\nClaude Code with `CLAUDE.md`\n|\n36.1% |\n| Claude Code with RE-call | 58.3% |\n\nRE-call improved success over the `CLAUDE.md`\n\nbaseline by 22.2 percentage points.\n\nThe 95% confidence interval was +11.1 to +33.3 points.\n\nThe paired McNemar test gave a p-value of 0.000145.\n\nThis was not one lucky demonstration. It was 72 paired comparisons across 24 tasks and three seeds, with zero discarded cells.\n\nEach task asked Claude Code to create or modify files in a repository and run the result.\n\nThe benchmark included tasks involving things like:\n\nThe important detail is that every task had a deterministic checker.\n\nThe agent could say \"done\" all it wanted. The checker only cared whether the repository ended up in the correct state.\n\nI ran every task through three configurations:\n\n`CLAUDE.md`\n\nThe harness also verified that the memory tools were actually available before counting a session. If the memory layer was missing, the entire paired cell was discarded.\n\nThat happened zero times in the DeepSeek run.\n\nThe most useful comparison is RE-call versus the static `CLAUDE.md`\n\nfile.\n\nThe static file produced a 36.1% success rate.\n\nRE-call produced 58.3%.\n\nThat is a 22.2 point improvement.\n\nThe paired results were even more revealing:\n\n`CLAUDE.md`\n\nlost`CLAUDE.md`\n\nwon only 1 comparison that RE-call lostThe effect was strongest on the tasks classified as memory-sensitive. Across those eight tasks, RE-call improved success by 45.8 percentage points.\n\nThat is the pattern I was looking for.\n\nMemory did not make every task easier. It helped when the task depended on a project decision, convention, or failure mode that was not fully present in the immediate prompt.\n\n`CLAUDE.md`\n\nThe static `CLAUDE.md`\n\nconfiguration performed worse than the bare configuration:\n\n| Configuration | Successful runs |\n|---|---|\n| Bare | 50.0% |\n`CLAUDE.md` |\n36.1% |\n\nThe bare configuration won by 13.9 percentage points.\n\nThis is not what I expected.\n\nIt suggests that a static instruction file is not automatically a memory system. It can add useful guidance, but it can also add noise, constraints, or instructions that compete with the task.\n\nRE-call is doing something different.\n\nInstead of putting every historical fact into the permanent prompt, it gives the agent a way to retrieve relevant project history when the current task needs it.\n\nThat distinction matters.\n\nThe benchmark also measured what happened inside the memory mechanism.\n\nIn the DeepSeek run:\n\nSo the result is not simply that \"the model had more tokens.\"\n\nThe memory layer was being consulted, and in most cases where it was consulted, it reached something useful.\n\nThere is still room to improve retrieval. A search is not automatically a good search. The model needs to find the right historical decision, not merely retrieve something related to the words in the prompt.\n\nThe complete DeepSeek run cost an estimated $0.4964 at the captured API prices.\n\nThe three arms cost:\n\n| Configuration | Estimated cost |\n|---|---|\n| Bare | $0.0824 |\n`CLAUDE.md` |\n$0.0863 |\n| RE-call | $0.3277 |\n\nRE-call used about four times as many total tokens as the static prompt configuration.\n\nThat is the tradeoff.\n\nThe memory layer costs more on every task, including tasks where memory is unnecessary. In exchange, it can prevent the much more expensive failure mode where an agent confidently follows an outdated convention, repeats a known mistake, or spends several minutes debugging a problem the team already solved.\n\nThe next engineering question is not whether memory helps.\n\nIt is how to make the retrieval smaller, faster, and more selective.\n\nI also started the same model-freeze run with GPT-5.3 Codex.\n\nThat run is not a negative result for GPT.\n\nThe provider hit a credit and in-flight request limit while the benchmark was running. The harness correctly discarded the affected paired cells instead of treating missing sessions as failed model attempts.\n\nOnly 40 of the 72 paired cells were admissible, so I am not using that run to claim that GPT-5.3 Codex is better or worse.\n\nThe descriptive numbers pointed in the same direction:\n\n| Configuration | Successful runs |\n|---|---|\n| Bare | 47.5% |\n`CLAUDE.md` |\n30.0% |\n| RE-call | 50.0% |\n\nBut the run needs to be repeated with the provider capacity issue fixed before it becomes a valid model comparison.\n\nThat is an operational failure, not a model conclusion.\n\nThis experiment does not show that memory makes an agent generally smarter.\n\nIt shows something narrower and more useful:\n\nA production memory layer can improve the probability that an agent completes a real repository task correctly.\n\nIn this run, RE-call improved success by 22.2 percentage points over the static project file.\n\nIt also improved success over the bare configuration, although that gain was smaller at 8.3 points. That difference is important. The main value is not simply adding more instructions. The value is retrieving the right project-specific context when it matters.\n\nThe first experiment showed that memory can make answers more correct.\n\nThis experiment showed that the improvement survives contact with the repository.\n\nThat is the result I was actually waiting for.\n\nThe next step is a clean competitor comparison:\n\nThe goal is not to produce another flattering demo.\n\nThe goal is to find out whether a memory layer remains useful when the models, vendors, and retrieval systems change.\n\nThat is the benchmark I want to run next.\n\n**Memory that abstains instead of guessing.**\n\nRE-call is agent memory on your own PostgreSQL with pgvector: every hit carries a verdict\nconfidence and provenance, a retracted claim comes back marked `superseded`\n\n, and a\nquestion the corpus cannot answer is refused rather than answered from the nearest neighbour\n\nATM-Bench Recall@10 **92.8924**\nagainst 79.09\nfor the best published row ([limits](https://github.com/GiulioDER/RE-call/blob/master/docs/ATM_BENCH.md))\n·\nsecond of ten on MTRAG correct refusals ([limits](https://github.com/GiulioDER/RE-call/blob/master/docs/MTRAG_BENCHMARK.md))\n·\nzero memory-layer LLM calls to build memory, where Mem0 pays one per session ([limits](https://github.com/GiulioDER/RE-call/blob/master/benchmarks/REVIEW.md))\n\n[Why RE-call](https://github.com/GiulioDER/RE-call#why-re-call)\n·\n[Quickstart](https://github.com/GiulioDER/RE-call#quickstart)\n·\n[How it works](https://github.com/GiulioDER/RE-call#how-it-works)\n·\n[Product surface](https://github.com/GiulioDER/RE-call#product-surface)\n·\n[Documentation](https://github.com/GiulioDER/RE-call#documentation)\n·\n[Evidence](https://github.com/GiulioDER/RE-call#evidence)\n\n[Setup guide: install, configure and run RE-call](https://giulioder.github.io/RE-call/)\n·\n[Validity Frontmatter: the open spec RE-call implements](https://github.com/GiulioDER/validity-frontmatter)\n\nNearest-match retrieval cannot tell the difference between what is true and what merely reads like it. When a corpus keeps its history, and real agent memory does, the retracted…", "url": "https://wpnews.pro/news/i-gave-claude-code-a-brain-this-time-i-let-the-tests-judge-it", "canonical_source": "https://dev.to/gde03/i-gave-claude-code-a-brain-this-time-i-let-the-tests-judge-it-5h0p", "published_at": "2026-08-24 16:06:44+00:00", "updated_at": "2026-08-24 16:13:44.389318+00:00", "lang": "en", "topics": ["artificial-intelligence", "ai-agents", "developer-tools", "machine-learning"], "entities": ["Claude Code", "RE-call", "DeepSeek"], "alternates": {"html": "https://wpnews.pro/news/i-gave-claude-code-a-brain-this-time-i-let-the-tests-judge-it", "markdown": "https://wpnews.pro/news/i-gave-claude-code-a-brain-this-time-i-let-the-tests-judge-it.md", "text": "https://wpnews.pro/news/i-gave-claude-code-a-brain-this-time-i-let-the-tests-judge-it.txt", "jsonld": "https://wpnews.pro/news/i-gave-claude-code-a-brain-this-time-i-let-the-tests-judge-it.jsonld"}}