Every era of software engineering builds its own vocabulary, and that vocabulary tells you what the era considered hard. Object orientation gave us reuse. Distributed systems gave us failure. Concurrency gave us safe parallel work. In agentic AI development, the new vocabulary is telling on itself: memory, loops, context window, tool use. Beneath it is a quieter shift. The bottleneck stopped being the problem and became the token.
Most conversations still ask whether a model can solve a task. For a long class of real work, it can. The real constraint is whether it can solve that task without drowning in everything it does not need to see right now. Context is not free. It is not neutral. A long-running agent accumulates yesterday’s dead ends, a schema that changed twice, and a conversation that already resolved itself. That junk costs exactly the same, token for token, as the thing that actually matters.
If you are building or operating agents across multiple sessions, this post is for you. If you have watched an agent re-read its own codebase from scratch at the start of every session, you already know the problem. If you have tried pasting hand-curated recaps until the human became the bottleneck, you know why that does not scale. What follows is a narrow, testable approach: treat memory as a budget to manage on purpose, measure whether it buys tokens back, and keep the implementation small enough to understand in an afternoon.
The strategy problem is repetition tax #
An agent with no shared memory is not helpless. It is expensive. Every fresh session starts cold. No memory of yesterday’s schema decision. No memory of why a function is shaped the way it is now. No memory of the bug already found and fixed once. The only way to recover that context is to either paste it back in manually, or have the agent re-read the codebase and rebuild its understanding from scratch.
Past a certain scale, neither path survives. Re-reading keeps getting more expensive because the project keeps growing. Every earlier session leaves files, tests, and decisions behind. By session six, the fifth session’s work is part of the environment the next session must absorb before it can do anything useful. Long context windows push the wall further out, but they do not remove it. The strategic question is not how much context the model can hold. It is which tokens earn their place there session after session.
This is where the terminology matters. Memory is not a chat log saved to disk. It is a budgeted retrieval surface. The interesting question is not whether the agent has records. It is whether those records stay small, ranked, and intentionally written.
A lightweight cross-session memory architecture #
The experiment behind this post used a small MCP memory server backed by SQLite, exposing a narrow set of tools: recall, remember, supersede, list_open, set_status, memory_history, and memory_stats. The underlying bet was specific: a project’s working knowledge is small, structured, and worth treating as data, not as a searchable chat log.
How the session boundary works
Most of what happens inside a session is reasoning noise. The conversation, the back-and-forth, the false starts and corrections, all of that gets thrown away when the session closes. The only artifact meant to survive is what was explicitly distilled into the store. That is different from dumping transcripts. It is closer to maintaining a project-level knowledge graph that an engineer would keep in their head if they were the only one returning to the code every day.
What survives the reset
The operational pattern is simple. Every session starts by recalling relevant prior memories before doing real work. Every meaningful work block ends with at least one explicit write back to the store, using a remember-style call for new facts and a supersede call when the old fact is now wrong. The end of the session is a hard reset. Conversation context is discarded. The next session reads only the SQLite-backed facts. Everything else starts fresh.
The recall and write cycle
The concept tested here treats memory as the third option: small, structured, ranked, and deliberately written. It pairs well with broader infrastructure work. If your team is already investing in AI, ML & CV development or internal MCP-backed persistence work, this is the kind of lightweight store that keeps those systems from burning context on repetition. For a broader systems view before implementing one, see the MCP Servers Comprehensive Handbook for CXOs.
The shape of the experiment matters more than one number #
To test whether any of this mattered, the same task ran twice. A five-rule discount pricing engine was built incrementally across six fresh sessions with no shared conversation state. After those sessions completed, a seventh audit session investigated a reported bug that touched every rule at once.
One track had the memory layer wired in. The other did not, and had to re-read its own codebase from scratch each session. Both tracks reached the same final answer: the reported bug did not exist under the rules as written, and the agent refused to invent a fix. That closure is not incidental. A cheaper but wrong answer would make the whole exercise a bad trade. Here, cheaper and correct arrived together.
The headline numbers can mislead you if you stop there. The no-memory audit session needed 8,150 tokens. The memory-enabled audit session needed 564 tokens. Across all six sessions, the memory track used 304,115 tokens and the no-memory track used 333,928 tokens. That is an 8.9% reduction across the full run, but most of the run was identical code-writing work in both tracks. The whole-run number is noisy.
The number worth tracking is context-relay cost: what each track needed to spend just to re-establish everything decided so far before starting the next session’s work. That cost behaves very differently over time. The memory track’s cost grows slightly, then caps and flattens because recall is built to stop filling at a fixed token budget no matter how much history exists behind it. The no-memory track’s cost grows linearly because every new rule adds more code that must be read in full. In any single session the difference looks modest. By session twenty, the curve matters.
This is exactly the kind of cost-control problem we write about in AI Agent Budgets and Cost Control and Production AI Agent Loops Engineering.
What the store actually saved #
Real memory content from the memory-enabled run, final state, was seven rows. That is not a simplification for storytelling. The distillation truly stayed that small. Rule four had no separate row because it did not add new behavior. It modified rule one’s exact logic, so its substance lived inside a supersession entry against rule one. That is the mechanism working as designed. When behavior genuinely changed, the store kept exactly one live, correct version, not two contradictory beliefs.
There was also a mid-experiment mistake worth citing. At one point, the wrong call used force=True, which would have created a duplicate instead of superseding. The error was caught by inspecting the database before it propagated further. That inspection step is straightforward. This is not theoretical correctness. It is operational discipline with a small, inspectable state store.
Why this is different from raw history or a smarter filesystem #
Cross-session memory for agents is not a new conversation. The approaches already on the table mostly fall into two buckets. The first bucket stores more raw history. The second builds a smarter filesystem with better search. Raw history does not solve the problem because verbosity is not the problem. Ranking and retrieval behavior is. A smarter filesystem does not solve it either, because files are not structured for the small, ranked, budget-enforced dataset that an agent should actually consult before doing work.
The concept tested here treats memory as the third option: small, structured, ranked, and deliberately written. It pairs well with broader infrastructure work. If your team is already investing in AI, ML & CV development or internal MCP-backed persistence work, this is the kind of lightweight store that keeps those systems from burning context on repetition. For a broader systems view before implementing one, see the MCP Servers Comprehensive Handbook for CXOs.
Where this breaks down today #
Honesty matters more than hype in an experimental system. There are real failure modes.
Isolation and subagent scope
Isolation is a configuration property, not a guarantee. In the first experimental run, subagents inherited the parent session’s MCP server scope and wrote into the wrong project’s store. MCP server scoping is inherited, not derived from the working directory. If you run subagents across multiple projects, you have to make isolation explicit.
Keyword-only retrieval
Retrieval is keyword-only. Ranking is BM25 multiplied by salience. There are no embeddings and no semantic fallback. A query phrased differently from how a memory was written will not match. That is a real ceiling on how much this approach can help when your project’s memories compress into abstractions you no longer search for literally.
Manual distillation
The memory layer does not extract decisions from a session automatically. It depends on the agent following an instruction to call remember() after meaningful work blocks. If that instruction slips, the store stays shallow. If the agent does, the store can make later sessions dramatically cheaper.
Budget cliffs
The budget is a cliff, not a slope. Once live memory content exceeds budget_tokens, recall() drops the lowest-ranked remainder. There is no tiered fallback, no graceful degradation, and no warning that material was removed. Scalable systems usually need softer fallbacks.
Salience without feedback
Salience measures recency and frequency. It does not measure whether a recalled memory actually helped. There is no feedback loop from task outcome back into ranking. A memory that gets retrieved often might be noise. A memory that stops a rebuild might never get ranked highly.
What to do with this next #
If you want to try this without building everything from scratch, the path is short. Install a lightweight SQLite-backed MCP memory server, run init inside your project, and point its viewer at a folder of projects afterward to see whether the savings hold on real work. If you already have an AI agent development effort underway, this is the kind of engineering decision we would test before committing to a heavier architecture. The problem this solves is not solved by larger context windows. It is solved by an external memory store that is cheap to query, cheap to write, and cheap to audit. The model does the reasoning. The store preserves only what should survive the session reset.
Memory earns its keep by staying small. That is the whole insight in one line. The best agents will eventually manage their own context, deciding on their own what matters and what to forget. We are not there yet. Until then, the practical move is to treat every token that survives a session as a deliberate outcome, not an accidental side effect.
This article originally appeared on lightrains.com
Leave a comment
To make a comment, please send an e-mail using the button below. Your e-mail address won't be shared and will be deleted from our records after the comment is published. If you don't want your real name to be credited alongside your comment, please specify the name you would like to use. If you would like your name to link to a specific URL, please share that as well. Thank you.