# Giving a fleet of AI agents one shared memory — when each agent runs a different model > Source: > Published: 2026-08-11 21:05:49+00:00 Most agent frameworks give each agent its own context window and call it memory. That works right up until you run more than one agent, and then it quietly becomes the most expensive design decision in the system. We run a fleet where different agents are deliberately backed by **different models** — one family handles long-form drafting, another handles structured extraction, a couple run on a local path with no external inference at all. Routing by capability is the easy part. The hard part is that an agent which learns something has learned it *alone*. This is a writeup of what broke, and the design we ended up with. The symptom shows up as repeated work. An extraction agent determines that a particular vendor's invoices put the tax line above the subtotal. Useful. Two days later a different agent — different model, different prompt, same pipeline — hits the same vendor and re-derives it from scratch. Then a third does it again. Nothing is *wrong*. Every agent behaves correctly. The system as a whole just has no way to accumulate anything, because knowledge lives inside whichever context window happened to be open at the time. You are paying inference costs to rediscover facts you already own. The naive fix is to pass more history. That fails for a specific reason worth naming: **context windows are per-invocation and per-model.** A 200k window on one model does not help an agent Once you accept that memory has to live *outside* the agents, the requirements get concrete: Point 3 is the one people skip, and it is the one that hurts. A shared memory store with no provenance turns every bad output into an unbounded investigation. Two layers, deliberately separated: **An append-only event log** is the source of truth. Every memory write is an event with the agent identity, session identity, channel, and timestamp attached. It is never mutated. If a fact turns out to be wrong, you append a correction — you do not edit history. This is the layer that makes point 3 and point 5 possible, and it is boring on purpose. **A derived index** is what agents actually query. It is rebuilt from the log, which means it is disposable. Change your embedding model, change your chunking, decide semantic search was the wrong call for a given path — rebuild the index, the log is untouched. The important property is the direction of the dependency. **The index depends on the log. Nothing depends on the index.** That is what lets you swap retrieval strategies without a migration, and it Our current implementation runs a document-indexing backend over configured paths with a scheduled embed cycle, plus session export with a retention window. The specific backend matters much less than the split — we have changed it once already and the log made that a non-event. The instinct is to make retrieval smarter. Better embeddings, reranking, hybrid search. In practice the wins came from **narrowing what is searchable per agent** before ranking anything. An agent asking about invoice formats should not be searching across support transcripts. Not for quality reasons — for correctness reasons. Cross-domain semantic neighbours are exactly the kind of plausible-but-wrong context that produces confident nonsense. Scope first, then rank. A small correctly-scoped candidate set beats a large well-ranked one, and it is dramatically cheaper. **We stored summaries too early.** Summarising a session into memory at write time felt efficient and destroyed the ability to re-derive anything later when we changed our minds about what mattered. Store the raw event; summarise at read time if you need to. **We under-specified identity.** Early on, "which agent wrote this" meant an agent *name*, which we then renamed. Use a stable identifier that survives renaming, and record the model separately — you will want to answer "did the model change when the quality dropped?" **We assumed retrieval failures were retrieval failures.** Most were scoping failures wearing a costume. Honestly: it depends on fleet size. With one or two agents, per-agent context is fine and this architecture is overhead you do not need. The crossover came for us somewhere around five or six agents sharing a domain, where the rediscovery cost and the "why did it do that" investigations started dominating. The clearest signal that you have crossed it: you find yourself explaining the same fact to different agents, or you cannot answer why an agent produced a given output without reading raw logs. Both are memory-architecture problems presenting as prompt problems. *We build and deploy governed AI agents for Singapore businesses at VYR — every agent runs with a human approval gate on any action that writes, sends or pays, plus a full execution log. If you are working on multi-agent memory, I would genuinely like to compare notes in the comments.*