18 Jul 2026
by flirp
After failing to reinvent the field of transformers, I decided to go in a new direction: the totally unsaturated field of memory. I know what you're thinking: where could such an original and monumental idea have come from?
Humans have been using memory for some time, with mixed results. I felt the field was ready for my implementation.
Remember this #
I built several small worlds governed by hidden rules and populated them with nonsense words.
The model would see examples such as:
gax ◇ yul = tes
I would then rename everything before the test. This prevented the model from retrieving the exact answer and, more importantly, made the benchmark look like proper science.
RAG with a probation department #
This was not a normal memory system. Normal RAG retrieves old chunks and puts them back into the prompt, which is roughly the cognitive equivalent of emptying a filing cabinet onto someone’s desk.
YAMS has a slightly more elaborate bureaucracy:
- Store the original events with their provenance.
- Compose them into a general rule offline.
- Test that rule on fresh worlds it has never seen.
- Promote it only if it survives.
- Retrieve the rule later by checking whether it explains the new task’s visible observations.
- Admit a small, bounded packet into the model’s workspace.
So the memory is not just retrieved. It is compressed, cross-examined, given references, and eventually allowed near the model under supervision.
The distinction matters because renamed symbols make old episodes difficult to reuse, while the structure behind them remains the same. A rule can transfer where a matching chunk cannot.
Haiku received one of four things:
- nothing;
- raw events selected by ordinary RAG;
- oracle RAG, which was handed the exact source events rather than having to find them;
- a short, validated rule extracted from those events.
The results:
| Memory | Score |
|---|---|
| Nothing | 30.6% |
| RAG with raw events | 19.4% |
| Oracle RAG with the correct raw events | 22.2% |
| Validated rule | 55.6% |
Even an oracle that knew exactly which experiences mattered could not make the raw events useful. Giving the model its actual memories made it worse than giving it amnesia.
Giving it a concise lesson almost doubled its score. It was also faster and used fewer tokens, an outcome I found suspicious because none of my systems had previously improved two things at once.
The rules were reusable rather than stored answers. When I taught two operations separately, Haiku recombined them on a new composite task and went from 16.7% with no memory to 100% with both rules. It also managed to write valid rules from its own experience for four of nine structures. Once externally validated, its home-made rules worked just as well as the algorithmic ones.
The model had not changed its weights. The useful capability lived in the memory pipeline around it, which is either exciting or an unnecessarily complicated way to invent revision notes.
Unfortunately, it worked #
I tried the normal ways of making the result disappear.
Generic encouragement did nothing. The wrong rule made performance worse. A contradictory rule also made performance worse. Giving it more tokens did not help.
The content of the memory genuinely mattered.
This was inconvenient because I had already prepared emotionally for another negative result.
The smarter model problem #
Then I gave the same memory to larger models.
Haiku improved by 38.9 percentage points. Sonnet improved a little, although not enough to rule out statistical mischief. Opus improved by exactly zero.
Opus produced byte-for-byte identical answers with and without memory.
My exciting new capability had become a decorative text file.
The explanation was annoyingly reasonable: Opus could derive the rule itself. Memory cannot teach a model something it already knows.
The verify-gap #
I needed information that Opus could not reconstruct, so I invented arbitrary obligations:
“When you later see trigger T, respond with R.”
The response appeared nowhere else. No amount of reasoning could recover it.
Without memory, every model scored 0%.
With memory, Haiku, Sonnet, and Opus scored 100%.
Even Opus cannot deduce something I made up five minutes ago. For now, this remains my comparative advantage.
Several completely normal research errors #
During the project:
- My first retrieval system achieved 100% accuracy by reading a hidden identifier.
- Removing that identifier reduced accuracy to 0%.
- A promising Sonnet result stopped being significant when I discovered an entire batch had not been saved.
- I built a benchmark with millions of apparent possibilities, which Opus solved as a small linear system.
- Some agents could use Bash when they were supposedly reasoning unaided.
Every mistake produced a plausible number. None displayed a warning saying FLIRP HAS ACCIDENTALLY PROVED HIS OWN SOFTWARE WORKS.
What survived #
Memory is useful when it contains something the model cannot reliably derive or verify.
This means agent memory should probably focus on private preferences, undocumented constraints, previous commitments, and arbitrary conventions, not facts the model can cheaply rediscover.
Unfortunately, unverifiable memories are also the dangerous ones. If memory falsely says you hate semicolons, the model cannot consult the universe for an independent answer.
So the final result is:
- Memory can give a frozen model a real transferable capability.
- Better models need it less when they can derive the answer.
- Every model needs it when the information is genuinely private.
- My retrieval system initially cheated.
Progress.
Code, experiment harness, and results: github.com/Oli-26/YAMS
Get the next one
New experiments, negative results included. No schedule, no spam, unsubscribe by replying.
Comments
Found a hole in this? Say so. Corrections and replications are the whole point, and a comment pointing at a mistake is worth more to me than a compliment.