Previous parts of Build a Basic AI Agent From Scratch:
You can find and clone this code in this blog series' Github repo.
In the previous parts we made our agent capable, autonomous and much safer. It can use tools in a Docker sandbox, plan long tasks, ask for help, respect a tool policy, and stop when it reaches a resource limit. But every time we close the program, it forgets everything.
This is a big limitation. The agent might learn that you prefer concise answers, that a project uses uv, or that a migration was partially completed yesterday. On the next session it has to discover all of that again.
In this part we will add long-term memory. Our agent will be able to save durable notes about the project and its user, then use them in future sessions.
The messages list in our agent loop is already a kind of memory. It contains the system prompt, the user messages, the model replies and the tool results. This is short-term memory: it lasts while the process is running and it is sent to the model as context.
It has two important limits:
Long-term memory is different. It survives across sessions and is retrieved only when it is useful. We will use it for two different things:
uv for Python commands".
Memory makes an agent more useful, but it also introduces a new privacy and safety problem: memory is persistent data. A bad or overly broad note can influence a later task long after the original context is gone.
For this first implementation, the system prompt gives the model strict rules:
"- Long-term memory (read_memory / append_memory): at the start of a session, read "
"both project and user memory. Project memory is for durable lessons, environment "
"facts, and conventions; user memory is for stable preferences and expectations. "
"Append only concise, useful facts that are likely to hold in future sessions. Never "
"store secrets, credentials, or transient task details.\n"
This is important for a few reasons:
By default, the memory directory is outside the Docker project mount. The agent accesses it through MemoryStore, not through its filesystem tools.
We do not need a database or vector store to start. A few files are enough, easy to inspect, and easy to back up. The default directory is ~/.agent-harness/memory:
~/.agent-harness/memory/
project.md
user.md
project.md and user.md contain append-only notes. With this design in mind, let's implement the memory store:
class MemoryStore:
"""Persist agent notes, user preferences, and complete session records."""
def __init__(self, root: Path):
self.root = root.expanduser()
self.sessions_dir = self.root / "sessions"
self.root.mkdir(parents=True, exist_ok=True)
self.sessions_dir.mkdir(exist_ok=True)
def read_notes(self, scope: str) -> str:
path = self._notes_path(scope)
return path.read_text(encoding="utf-8") if path.exists() else "(empty)"
def append_notes(self, scope: str, content: str) -> str:
path = self._notes_path(scope)
with path.open("a", encoding="utf-8") as file:
if path.stat().st_size:
file.write("\n")
file.write(content.strip() + "\n")
return f"Saved {scope} memory."
The _notes_path helper only accepts project and user. It doesn't let the model choose an arbitrary filename:
def _notes_path(self, scope: str) -> Path:
if scope not in {"project", "user"}:
raise ValueError("scope must be 'project' or 'user'")
return self.root / f"{scope}.md"
The agent needs tools to access this store. All of them run in-process, alongside the scratchpad and to-do list, because they hold harness state rather than acting on the project files:
def read_memory(scope: str) -> str:
return _get_store().read_notes(scope)
def append_memory(scope: str, content: str) -> str:
return _get_store().append_notes(scope, content)
To work with long-term notes, the agent gets two tools:
read_memory: read project or user notes.append_memory: append a concise durable note.
These tools are in PLANNING_TOOLS, so they do not prompt for permission on every call. That is a reasonable default because they only access the configured memory directory. It also means the memory directory should be treated as sensitive user data and protected with normal operating-system permissions and backups.
First, we can ask the agent to remember a stable preference:
$ uv run agent.py
Session: z7Pq3hM4kA
Memory directory: /home/xxxxx/.agent-harness/memory
You: I prefer concise answers. Remember that for future sessions.
[tool] read_memory({'scope': 'user'})
[tool result] (empty)
[tool] append_memory({'scope': 'user', 'content': 'The user prefers concise answers.'})
[tool result] Saved user memory.
Assistant: I will keep answers concise in future sessions.
Next time, the agent can read that note before it answers:
You: Help me understand this project.
[tool] read_memory({'scope': 'project'})
[tool result] Use uv for Python commands.
[tool] read_memory({'scope': 'user'})
[tool result] The user prefers concise answers.
Assistant: I will inspect the project and keep the explanation concise.
Our agent now has durable memory without turning its whole history into every model request:
The agent can now keep useful facts between runs, so it does not have to rediscover the same preferences and project conventions every time.
This article introduced the foundations for sessions. In the next article, we will build on them: listing past sessions, resuming an unfinished session, and compacting a long session so it can continue within the model's context window.