How Much Memory Does Your Agent Need? — A Practical Memory Store Selection Guide

An engineer's practical guide to selecting memory stores for AI agents argues that vector databases are often overkill, citing that 95% of agent time in a university admissions scraper was spent on simple state reads and that vector retrieval accounted for only 3.7% of memory requests at ByteDance's internal agent platform. The article proposes a three-layer memory architecture using built-in key-value storage, Markdown files, and error logs to cover session state, domain knowledge, and error history without adding dependencies.

The Pain: You search GitHub and find everyone using different memory stores — ChromaDB, PostgreSQL, plain Markdown files, "SQLite is good enough for a decade." Which is right? The Answer: All of them. And none of them.Choosing without understanding your scenario is like buying a car without checking the road. When I was building a university admissions data scraper 91 universities , I made the classic mistake: I equipped the agent with a full ChromaDB + vector retrieval stack, spent three days tuning it, and then discovered — 95% of the agent's time was just reading "which page did I get to last time." A boolean would have sufficed. I built a vector database capable of semantic search. An engineer friend at ByteDance told me their internal agent platform found, after six months, that vector retrieval accounted for only 3.7% of all Memory Store requests. The remaining 96.3%? Key-value lookups, state reads/writes, error dedup. The vector search you spent two weeks integrating might serve less than 4% of your queries. So before discussing "which storage," we must answer a more fundamental question: what does your agent actually need to remember? I categorize memory into four types: | Memory Type | Typical Content | Size | Access Frequency | Consistency | |---|---|---|---|---| | Session state | "Processing university 37/91" | ~100 bytes | Every call | Strong | | Domain knowledge | "A-University rate limit is 10 req/s" | ~1KB | On demand | Eventual | | Error history | "B-University returns 403 because UA blocked" | ~MB | Before new task | Append-only | | Semantic memory | "Map 'that red button' to settings page" | varies | Occasional | Eventual | See the pattern? If your agent mainly does multi-step automation data scraping, report generation, CI/CD pipelines , the first three types are the real needs — and none of them require a vector database. Core thesis: memory selection is about finding the layer that is "just enough." One layer too many is waste; one layer too few is disaster. I've used each of these across three projects of different scales. The "crash moments" are recorded in the table: | Dimension | memory /STATE.md | SQLite | ChromaDB | PostgreSQL | |---|---|---|---|---| | Type | KV memory+JSON | Markdown file | Embedded relational | Vector DB | | Capacity | ~4,000 chars | No hard limit | TB-scale | TB-scale | | Query | Exact key match | grep/regex/fulltext | SQL JOIN/aggregate | Semantic top-K | | Install | 0 built-in | 0 OS | pip install | pip + ONNX/API | | Ops | None | None | Low single file | Medium index | | Concurrency | Single process | Lock-free | WAL mode | Single writer | | Latency | <1ms | <1ms | <5ms | 10-100ms | | Best for | Preferences/contracts | Project progress/state | Structured task data | Long-tail semantic search | Crash moments real experience : Three layers complement each other: L1 fast but small / L2 structured / L3 dedupable. Here's an architecture validated across three projects. No new dependencies — just a combination that covers "session state + domain knowledge + error history." ┌──────────────────────────────────────────┐ │ Three-Layer Memory Architecture │ │ │ │ L1: memory ← 4KB KV, preferences │ │ L2: STATE.md ← file, progress │ │ L3: ERROR LOG.md ← file, pitfalls │ │ │ │ L1 fast but small / L2 structured / │ │ L3 dedupable │ └──────────────────────────────────────────┘ Best for: user preferences "reply in Chinese" , project conventions "outputs go to /outputs/" , environment facts "Python 3.11" . Fatal trap : the 4,000-char hard limit. The solution isn't to avoid it — treat L1 as a cache and push critical data down to L2. The backbone of three-layer memory. A single Markdown file, read at session start, updated after task completion. Core code is under 80 lines: state manager.py — automatic STATE.md reader/writer """Agent's workbench — read each session, update after each task""" from pathlib import Path from datetime import datetime import re STATE FILE = Path "./STATE.md" def load state - str: """Call at session start: read STATE.md, inject into agent context.""" if STATE FILE.exists : content = STATE FILE.read text now = datetime.now .strftime "%Y-%m-%d %H:%M" return f" Project current state last updated: agent-maintained \n" f" Time: {now}\n\n{content}\n\n" f"---\nAbove is project state. Continue based on this.\n" First use: initialize template return """ Project State File Maintained by agent. Read at each session start. Active Tasks none Blockers none Attempted & Failed none Project Overview | Metric | Value | Note | |--------|-------|------| | Status | OK | - | """ def update state section: str, content: str : """Call when agent completes a task or finds a blocker.""" current = STATE FILE.read text if STATE FILE.exists else load state marker = f" {section}" if marker in current: parts = current.split marker, 1 before = parts 0 after parts = parts 1 .split "\n ", 1 rest = "\n " + after parts 1 if len after parts 1 else "" new content = before + f"{marker}\n{content}\n" + rest else: new content = current.rstrip + f"\n\n{marker}\n{content}\n" STATE FILE.write text new content print f" state manager Updated section '{section}'" Append-only log of errors with dedup. The key mechanism: each error gets a fingerprint e.g., hash of the error message . If the same fingerprint appears, skip it — no duplicate entries. python error log.py — deduplicated error logging from pathlib import Path import hashlib from datetime import datetime ERROR FILE = Path "./ERROR LOG.md" def log error scene: str, message: str, solution: str = "" : """Log an error, dedup by message hash.""" fp = hashlib.md5 message.encode .hexdigest :8 current = ERROR FILE.read text if ERROR FILE.exists else "" if f" {fp} " in current: return already logged — skip entry = f"\n {fp} {scene} @ {datetime.now .strftime '%Y-%m-%d %H:%M' }\n" f"- Error: {message}\n" f"- Solution: {solution or 'TBD'}\n" with ERROR FILE.open "a" as f: f.write entry Three layers complement each other: L1 fast but small / L2 structured & readable / L3 dedupable & queryable. Need to store... ├── User preference / project convention? → L1 memory ├── Task progress / state? → L2 STATE.md ├── Error / pitfall? → L3 ERROR LOG.md ├── Structured task data queries needed ? → SQLite ├── Semantic search? → ChromaDB only if 4% case └── Multi-tenant production? → PostgreSQL Rule of thumb: start with the simplest layer that works. Add complexity only when you hit a concrete limit. You're no longer the developer who installs a vector database because "it's what the cool kids use." You're becoming an architect who matches storage to the actual memory pattern of your agent. Most agents don't need vector search. They need a sticky note, a navigation map, and a pitfall record. The three-layer architecture gives you all three with zero new dependencies. Next: Quality isn't accidental — Maker/Checker separation and automated validation. About the author: Wu Ji 无记 — AI & digitalization practitioner focused on Agent engineering, Loop Engineering, and digital transformation. Practical, hands-on tutorials — follow along and it just works.