A coding agent starts each session cold. It has no idea what you decided last Tuesday, why you rejected the obvious approach, or which config value burned an afternoon. You can paste the context back in every time, or you can give the agent a memory it can query.
The second option sounds simple until you write it down as an engineering problem, because it has two constraints that pull against each other:
This post is about how we built that layer for Kireo, and — more usefully — the things that broke along the way. Honest framing up front: this isn't a scale-bragging post. It runs on one small VM, and the interesting part is how far a modest box goes when the retrieval path stays lean.
The most common pushback on any agent-memory idea is: isn't this just going to bloat every prompt? Fair worry, and the answer is entirely in the design.
Memory is an MCP tool the agent calls on demand, not a blob injected into every turn. When the agent needs prior context it calls memory_search
, gets back a small ranked set of hits (default 10, hard cap 50), and spends tokens only on those. Nothing is prepended to the system prompt; nothing runs on turns where the agent doesn't ask.
That distinction — pull, not push — is the whole reason the token math works. A per-turn injection scheme pays for memory on every message whether it helps or not. An on-demand top-k tool pays only when the model judges the context worth retrieving, and the budget is bounded by limit
.
The server ships eight tools over MCP stdio — memory_save
, memory_search
, memory_recall
, memory_get
, memory_update
, memory_delete
, memory_list_namespaces
, and memory_health
— and every MCP client (Claude Code, Cursor, and others) sees the same set. But memory_search
is the one that has to be both fast and frugal, so that's where the engineering went.
The whole retrieval path:
intfloat/multilingual-e5-small
, a 384-dimension model, on CPU. It speaks an OpenAI-compatible /v1/embeddings
endpoint on the internal network.Two of those choices are the load-bearing ones.
LanceDB is embedded, not a service. For a workload this size, a managed vector database is overkill. LanceDB is columnar on disk, the query path is a library call, no network hop. The vector column is an Arrow FixedSizeList(dim)
— remember that, it comes back to bite us.
384 dimensions on CPU is enough, and it's cheap. Memory snippets are short — a decision, a gotcha, a config note. You don't need a 1536- or 3072-dim frontier model to separate "we chose Postgres row-level security over app-layer checks" from "the CI cache key needs the lockfile hash." Smaller vectors mean cheaper ANN and less storage, and a small e5 model does short-text semantic matching well on CPU. The one catch: e5-family models want asymmetric prefixes — passage:
for stored documents, query:
for search queries — supplied from config and empty for OpenAI-style models. (That innocuous trailing space caused a real bug; more below.)
A side effect of TEI's OpenAI-compatible endpoint: the same client code talks to the self-hosted model by pointing a base URL at the internal service — swapping providers was a config change, not a rewrite. The cache and rate-limit layers use the same trick: a self-hosted Redis fronted by a shim speaking the Upstash REST API, so the REST client needs no managed account.
Pure vector search misses exact-match cases (a specific error code, a function name). Pure keyword search misses paraphrase. So memory_search
runs both and fuses them.
The flow:
RRF is deliberately boring: it combines rankings by 1/(k + rank)
without needing the two scoring systems to share a scale — cosine distance and BM25 don't. Robust and cheap, which is what you want in the hot path.
The interesting part is the degradation ladder: in a lean stack any dependency can be briefly unavailable, and search still has to return something:
None of that is glamorous, but it's the difference between "memory occasionally returns fewer hits" and "memory throws inside the agent's tool loop."
Every row carries a user_id
and a namespace
(e.g. code-my-app
for an indexed repo). Isolation is a user_id
predicate pushed into every LanceDB query, plus a belt-and-suspenders assertion after results return: if any row's user_id
doesn't match the caller, the code throws instead of leaking it. The filter should never be wrong — so we check anyway, on every read path. Cross-tenant leakage is the one bug you never want to ship, and a three-line assertion is cheap insurance.
The stack didn't start at 384 dimensions. It started at 1536 against a hosted model; moving to the self-hosted 384-dim e5 model meant every stored vector was now the wrong length.
Here's where LanceDB's FixedSizeList(dim)
schema stops being an implementation detail. The vector column's dimension is baked into the table schema, and in the version we run there is no in-place column resize and no table rename. Once the dimension changes, every insert fails against the old table, and you can't quietly widen the column.
The migration is dump-drop-recreate:
memories
table.embedding = null
and embedding_status = 'queued'
, then reset the Postgres status rows to queued
too so the backfill job re-embeds everything with the new model.The migration is idempotent — if the table is already at the target dimension it's a no-op — which matters when you're running it by hand on a live box, unsure whether the last attempt finished.
And there's a subtle second-order bug this exposed. The embedding cache is keyed by model + content hash — but not by dimension. So after a same-model endpoint change that alters the vector length, a stale cached vector of the old length would sail past the cache lookup and get written into the freshly recreated table, breaking the insert. The fix is a dimension guard at two layers: the embed client asserts vector.length === EMBEDDING_DIM
before returning (a wrong-length embed fails and degrades to the queue instead of corrupting the table), and the cache read treats a length mismatch as a miss. The rule: never let a wrong-dimension vector reach the table, enforced at every point one could enter.
Indexing a repo uploads symbols in batches — up to 100 per request. Batches time out sometimes, and the obvious retry (re-send the batch) creates duplicates unless the write path is idempotent.
The fix is content-hash dedup as a single set query, not N point lookups. Before inserting a batch, one query fetches the active (non-deleted) rows whose content_hash
is in the batch's hashes — content_hash IN (...)
for the whole batch — and skips them. Re-running the same upload is safe: identical content hashes to identical rows, retries don't multiply.
Two things I like here. First, one IN
query keeps dedup off the per-item hot path — one query per batch, not one per symbol. Second, it's the same guarantee surfaced in the CLI docs: if a batch upload times out, re-running the same command is safe. Not an internal nicety but a documented contract — the person hitting the timeout is the one who needs to trust the retry.
Delete is where naive implementations quietly lose data or lie about counts. Our fixes here were all, at heart, about semantics.
Deletes are soft by default: a delete stamps deleted_at
and sets a 30-day expires_at
restore window. The row stays in the table, filtered out of normal reads by deleted_at IS NULL
. Restore checks the window and refuses if it's expired; a TTL sweep physically removes rows past expires_at
.
That created three follow-on requirements that each needed explicit handling:
total − active
, because off-by-a-little count math is exactly what users notice and stop trusting.update
, so there's no window where the row doesn't exist.One more LanceDB-shaped wrinkle: a plain scan has no ORDER BY
. To list newest-first without materializing a million rows, the list path streams every matching batch, re-sorting as they arrive and truncating to limit + 1
so only the current top page stays in memory. It's more code than ORDER BY ... LIMIT
, but it's what the storage engine actually supports.
Three more that belong in any honest "one VM" story:
:latest
pull can't resurrect the bug.passage:
prefix and quietly degraded recall until the value was quoted. The kind of bug that has no stack trace.None of these are clever. They're the tax for running a real workload on modest hardware — and writing them down means the next person (often me, three months later) skips the debugging.
This is the memory layer behind Kireo, an MCP server that gives Claude Code, Cursor, and any MCP client one shared long-term memory — save decisions and gotchas as you work, recall them from any tool, and browse, edit, or delete everything in a web dashboard, with full JSON export if you ever want to walk away with your data. It's in free beta right now with generous limits and no card required.
Install is one line:
claude mcp add kireo --scope user --env KIREO_API_KEY=ki_sk_xxx -- npx -y --package=@kireo/mcp-server kireo-mcp
Grab a key and read the docs at kireo.app. On privacy: the server only sends what you explicitly pass to memory_save
; it never reads your code, and code indexing stores derived embeddings and file paths, not your source — see the data-storage policy. The MCP server is on npm and GitHub.
I've kept latency claims out on purpose — I'd rather ship measured numbers than round ones, and a follow-up profiling the search path is on the list. If you build agent memory on a small stack and hit a different set of walls, I'd like to hear which ones.