I'm writing a series called "Becoming a Super Engineer": a software engineer since 2013, learning to hold software, data and AI under the same belt, and writing it down as it happens.
Chapter 1 ended with agents that only stopped repeating mistakes once a check failed on them. This chapter is about the other half of that problem: what agents remember. It's about an open-source project of mine, a knowledge base the agents read before touching anything and write back to when they're done. It taught me a lot about working with agents :)
Two agents without memory, two different correct APIs. The one with memory gave back the fix we had recorded
Without persistent memory, an AI coding agent forgets everything between sessions.
Monday it spends 20 minutes figuring out how the auth middleware works. Tuesday, same question, same 20 minutes. Wednesday it debugs a bug you already fixed last week, from scratch.
My first pitch for a knowledge base was the obvious one: stop paying tokens to rediscover the same things.
A small experiment changed my mind. More on that later.
The real problem showed up in an experiment I had an AI agent run: three agents got the same money-rounding bug, in the same codebase.
All three fixed it correctly.
The two without the knowledge base came back with two different correct fixes. One added round_money(amount). The other added to_decimal, round_money(value) and percent_of(amount, pct).
The agent with the knowledge base rebuilt the fix we had already recorded: same API, same four call sites, same line numbers.
All three were right. Only one matched what the team already had.
That's the actual problem: drift. Each of these agents was smart enough to solve the problem, and each one solved it a little differently. Over months, that's how a codebase ends up with three ways to round money.
Each step moved a rule from "please do this" to "this fails if you don't"
The project is called knowledge-database. The shape is simple. A folder of dated markdown entries, in four buckets:
explorations/: what is true about this system solutions/: what we did, and how errors/: what broke, and the trap that catches it next time decisions/: what we chose, what we rejected, and who decided
Every entry cites its evidence as path:Lstart-Lend, so a claim points at real lines of code.
And one loop, on every non-trivial task: read first, do the work, write back.
Read first, do the work, write back. The check makes sure nobody skips step three
I'd been using this idea heavily in another setup before it went public. The public version, and making agents actually follow it, came together in twenty days.
The first versions were rules written in the agent instructions. Please read the KB first. Please record what you learned.
Recording started as a prompt at the end of every task ("want me to record this?"), and turned mandatory pretty quickly. I wanted to review what went into an entry, without approving every single one.
Everything in this phase depended on the agent choosing to behave.
Then someone reviewed it from the outside and showed me the conventions were drifting. Index tags didn't match the entries. Entries marked verified had prose instead of proof. One decision used a hypothetical code comment as its evidence.
That review changed the project. The new rule was simple: enforce or delete.
Every hard rule got a stable ID (KB001 to KB015 today) and a test fixture that passes or fails. A rule without an ID isn't a rule anymore. It moves to a "guidance, not checked" section.
A few of my favorites:
verified requires a verification block with captured output. Comments like # should work get rejected
The checks run from a zero-dependency Python CLI, a pre-commit hook, CI, and a hook at the end of every agent session.
A field failure showed the rules only loaded for one agent runtime, and only at the start of a session.
So now a short rules block (about 240 tokens) gets injected on every single prompt, and the same rules get planted in six agent instruction files, one per runtime. Install it mid-session and it works on the very next prompt.
Someone installed it fresh, on a workspace that wasn't a git repo, and read the docs like a new user. Thirteen findings.
The verdict hurt a little: everything mechanically enforced worked. Everything at the edges didn't.
The installer died halfway on a non-git folder. The rules told agents to run two scripts no install had ever copied. A staleness rule would start blocking every commit, 90 days after the entries were written, all on the same day.
All thirteen got fixed in one release, plus a setup wizard and a real upgrade path.
Then, in a second experiment, I had an agent measure the pitch itself. The control was the same task run three times without the knowledge base, recording each answer, the tokens and the time. Then the same task with the knowledge base in place.
The agent with the knowledge base cost more than the control.
Both causes were my fault. Nothing told the agent what "done" looks like when the answer already exists, so it wrote a new entry anyway. And the documented way to look things up was "read the index", which cost thousands of tokens before reaching the answer.
The fixes:
kb find "<keywords>" prints only the hits: 134 to 360 tokens instead of 2,360 to read the indexkb show <entry> prints just the useful part of one entry: 279 tokens instead of 709
Together, reusing an answer went from about 3,000 tokens to about 400 to 650.
And I rewrote the README. The old headline promised token savings that the measurements didn't support at small scale. Token savings still happen, but only when an answer gets read many times, or when the investigation behind it was expensive. The new headline is: your agents give the same answer twice, and can show their work.
Profiling found one more surprise: about 55ms of a 58ms lookup was just Python starting up. So now one command returns the hits and the top fix together, and the search reads a single index file instead of every entry.
My favorite bug came last. kb find returned the right entries, then added a note at the bottom: "(deep scan: no title/tag/summary match)".
An agent read "no match" as the verdict, ignored the verified entries listed right above it, and went off to re-derive the answer, which is exactly what the tool exists to prevent.
The fix was one line of text. The note now only says where the match came from.
If you want to try this, with my tool or your own, this is what I'd keep:
path:line. Explorations, solutions, errors, decisions. Every error entry records what broke and the trap that catches it next time.
And be honest about where it doesn't pay off: when the fact is one grep away, when the project is solo and short-lived, or when the code changes faster than you can re-verify entries. Stale memory is worse than none.
Chapter 2 lived mostly in the AI circle. Still heading for the middle :)
Looking back, almost every release came from outside feedback. A reviewer, a field report, a fresh install, an experiment, a profiler, an agent misreading my own tool.
So thank you to everyone who reviewed it, installed it fresh and broke it. It didn't always feel great in the moment, and it made the tool a lot better :)
And the direction never changed: take something that depends on an agent choosing correctly, and turn it into something that fails loudly instead.
That's the same lesson from Chapter 1, just one level up. There it was deploy traps. Here it's knowledge itself: a claim with a test attached holds up much longer than one nobody checks.
For the Super Engineer path, this was the AI side getting real for me. I think a big part of the work there is designing what the agent can't skip.
If you're feeling the AI fear right now, I believe this is a good place to start: you already know where your team's traps are. Write them down in a way an agent has to respect.
The project is open source: [github.com/dcoferraz/knowledge-database](https://github.com/dcoferraz/knowledge-database). Issues and stories very welcome.
Let me know in the comments: how do your agents remember things today? Or do they?