Build a Basic AI Agent From Scratch: Memory

The "Build a Basic AI Agent From Scratch" tutorial series added long-term memory to its agent harness, storing append-only notes in project.md and user.md files under the default directory ~/.agent-harness/memory. The implementation uses a MemoryStore class whose _notes_path helper accepts only the scopes "project" and "user", with read_memory and append_memory tools running in-process and the memory directory kept outside the Docker project mount. The series frames persistent memory as a privacy and safety concern, instructing the model never to store secrets, credentials, or transient task details.

Previous parts of Build a Basic AI Agent From Scratch : You can find and clone this code in this blog series' Github repo https://github.com/rogiia/basic-agent-harness . In the previous parts we made our agent capable, autonomous and much safer. It can use tools in a Docker sandbox, plan long tasks, ask for help, respect a tool policy, and stop when it reaches a resource limit. But every time we close the program, it forgets everything. This is a big limitation. The agent might learn that you prefer concise answers, that a project uses uv , or that a migration was partially completed yesterday. On the next session it has to discover all of that again. In this part we will add long-term memory . Our agent will be able to save durable notes about the project and its user, then use them in future sessions. The messages list in our agent loop is already a kind of memory. It contains the system prompt, the user messages, the model replies and the tool results. This is short-term memory : it lasts while the process is running and it is sent to the model as context. It has two important limits: Long-term memory is different. It survives across sessions and is retrieved only when it is useful. We will use it for two different things: uv for Python commands". Memory makes an agent more useful, but it also introduces a new privacy and safety problem: memory is persistent data. A bad or overly broad note can influence a later task long after the original context is gone. For this first implementation, the system prompt gives the model strict rules: "- Long-term memory read memory / append memory : at the start of a session, read " "both project and user memory. Project memory is for durable lessons, environment " "facts, and conventions; user memory is for stable preferences and expectations. " "Append only concise, useful facts that are likely to hold in future sessions. Never " "store secrets, credentials, or transient task details.\n" This is important for a few reasons: By default, the memory directory is outside the Docker project mount. The agent accesses it through MemoryStore , not through its filesystem tools. We do not need a database or vector store to start. A few files are enough, easy to inspect, and easy to back up. The default directory is ~/.agent-harness/memory : ~/.agent-harness/memory/ project.md user.md project.md and user.md contain append-only notes. With this design in mind, let's implement the memory store: class MemoryStore: """Persist agent notes, user preferences, and complete session records.""" def init self, root: Path : self.root = root.expanduser self.sessions dir = self.root / "sessions" self.root.mkdir parents=True, exist ok=True self.sessions dir.mkdir exist ok=True def read notes self, scope: str - str: path = self. notes path scope return path.read text encoding="utf-8" if path.exists else " empty " def append notes self, scope: str, content: str - str: path = self. notes path scope with path.open "a", encoding="utf-8" as file: if path.stat .st size: file.write "\n" file.write content.strip + "\n" return f"Saved {scope} memory." The notes path helper only accepts project and user . It doesn't let the model choose an arbitrary filename: php def notes path self, scope: str - Path: if scope not in {"project", "user"}: raise ValueError "scope must be 'project' or 'user'" return self.root / f"{scope}.md" The agent needs tools to access this store. All of them run in-process, alongside the scratchpad and to-do list, because they hold harness state rather than acting on the project files: php def read memory scope: str - str: return get store .read notes scope def append memory scope: str, content: str - str: return get store .append notes scope, content To work with long-term notes, the agent gets two tools: read memory : read project or user notes. append memory : append a concise durable note. These tools are in PLANNING TOOLS , so they do not prompt for permission on every call. That is a reasonable default because they only access the configured memory directory. It also means the memory directory should be treated as sensitive user data and protected with normal operating-system permissions and backups. First, we can ask the agent to remember a stable preference: bash $ uv run agent.py Session: z7Pq3hM4kA Memory directory: /home/xxxxx/.agent-harness/memory You: I prefer concise answers. Remember that for future sessions. tool read memory {'scope': 'user'} tool result empty tool append memory {'scope': 'user', 'content': 'The user prefers concise answers.'} tool result Saved user memory. Assistant: I will keep answers concise in future sessions. Next time, the agent can read that note before it answers: You: Help me understand this project. tool read memory {'scope': 'project'} tool result Use uv for Python commands. tool read memory {'scope': 'user'} tool result The user prefers concise answers. Assistant: I will inspect the project and keep the explanation concise. Our agent now has durable memory without turning its whole history into every model request: The agent can now keep useful facts between runs, so it does not have to rediscover the same preferences and project conventions every time. This article introduced the foundations for sessions. In the next article, we will build on them: listing past sessions, resuming an unfinished session, and compacting a long session so it can continue within the model's context window.