# Agent Swarms Are a Distributed Systems Problem > Source: > Published: 2026-08-27 21:33:20+00:00 # Agent Swarms are a Distributed Systems Problem[#] *Authored by Robert Escriva* *Authored by Robert Escriva* Agents today are primarily single-player. With coding agents, a software engineer nudges along one agent toward a goal. When the engineer wants to work on multiple issues at once, they often reach for git worktrees to isolate each agent while it works. But the deferred conflicts all come due at merge---so in effect the swarm is editing one shared state, just with the collisions postponed. Shared state must solve for the conflicts that arise with concurrent writers. Git merge can work well for code because code is written in lines: line-based diffs mostly coincide with syntactic units, edits are mostly local, and there's an intelligence standing by to escalate conflicts to. But these assumptions do not hold for agent-written knowledge. Agent-written knowledge, like that recorded in a wiki or knowledge base, is based upon natural language; prose gets summarized, reorganized, rewritten, and retracted. An early version of Foundation did exactly this---git for conflict resolution---and it threw away more finished work than anything else we tried; we return to why below. Foundation, Chroma's solution to memory, operates through a swarm of agents modifying shared state. Foundation must also solve for concurrency control, and specifically we want to optimize for goodput---the share of paid reasoning that survives rather than being thrown away and re-done on abort. As we will show below, common approaches like leveraging database transactions fall short. The trap they share: on abort each one discards the paid-for reasoning and pays to redo it---and for an agent, redoing is minutes of latency and dollars of tokens. **Our specific workload** | Foundation | DB Transactions | | |---|---|---| | Work | Expensive to retry | Cheap to retry | | Length | Minutes | < Second | | Retry | Re-reasons from scratch | Re-runs same code | The first row is the crux: when a database transaction aborts, retrying re-runs the same code in milliseconds. When an agent's transaction aborts, the read set was discovered by searching and reasoning---so retrying means paying for that discovery all over again, in minutes and tokens. This was the biggest challenge to solve when building Chroma's new memory layer. Foundation is Chroma's new memory layer. Foundation ingests your coding agent traces and company data to build a durable record and index. At a high level, Foundation is powered by an agent swarm that cooperatively builds a wiki. Cooperation between threads with their own agenda is categorically a concurrency control problem, ported to the domain of agents instead of code. But a swarm of agents editing concurrently is something that textbook concurrency control is ill-equipped to handle. The system processing incoming data breaks it into batches of data to be processed incrementally by the swarm. Each agent takes a batch of traces and does the following: Search for relevant pages on the wiki, read the pages that seem relevant in the search results, and update those that would benefit from information in the batch. Everything is powered by a lightweight reasoning model. Before we dive into the system, it's important to know the baseline guarantees made by Chroma Cloud, the distributed storage system that powers Foundation. Chroma Cloud is *also* a distributed system and it supports transactions---added for Foundation storage. Foundation stores its chunked wiki pages in Chroma as the system of record. Chroma's optimistic concurrency control (OCC) for transactions make it possible to atomically update these chunks so the page is never shown in a partially written state. The OCC guarantees of Chroma Cloud's transactions are enough of a primitive to build any concurrency control system on top of Chroma, and yet, as we shall see, they alone are insufficient as the basis of Foundation's long-lived transactions. Two observations drive everything: The reason textbook approaches are insufficient follows straight from our observations above. Because the transactions are long-lived, OCC will not work; the transaction runs for minutes and validates at commit---the perfect retry storm. Because the read set is discovered through search, it is possible to acquire locks incrementally in a way that yields a cycle. Deadlock prevention techniques roll back transactions and discard paid-for tokens. Once you realize transaction abort is the problem, the rest follows, and early commit emerges. Consider what happens during an abort. Work that was done and tracked during the body of the transaction is inherently discarded. Work that was perfectly acceptable at the time gets discarded as well. A fact is no *less* relevant at the end when the transaction aborts than it was at the time the reasoning model put the transaction on the page. Contention, not transaction content or consistency guarantees, drives the abort. We embrace this observation within Foundation by leveraging the assumption that atomicity across the wiki as a whole is an explicit non-goal of Foundation. Our goal is to index source data on the wiki; reverting a valid edit to the wiki is strictly more expensive than leaving it. Individual wiki page writes remain atomic. Our protocol, dubbed Fission because breaking atomicity is encouraged, looks a lot like a two-phase locking protocol with incremental lock acquisition and wound-wait deadlock prevention. In a wound-wait system, deadlock is impossible. When a request would take a lock that would create a cycle, the younger transactions wait for older ones to complete, while the older transactions wound/abort newer transactions by aborting them to forcibly pre-empt the lock holder and prevent a deadlock cycle. The difference between fission and traditional two-phase locking is that it explicitly breaks atomicity; transactions never roll back. This may seem like a return to the days of eventual consistency and the CAP theorem as alternatives to ACID, but the reality is different. What Fission guarantees: The actual protocol is nearly textbook pessimistic concurrency control with wound-wait, but with a critical change: There is no "roll back on abort." Killing a transaction leaves exactly what the transaction already did because it is early-commit. Aborting is treated as early commit, not roll back. To summarize the protocol: That's the whole protocol. Of course, every new protocol has its trade-offs. Fission makes the assumption that incremental progress on the wiki is important to retain, and retrying a batch that gets wounded is sufficient to make forward progress. Wounding can and does still happen in Fission. A wounded transaction leaves its work behind for others to discover. Because of the locking protocol, isolation is afforded to those discoverers, too. In our experience, giving the agent a TODO tool structures the writes in a way such that every prefix is valid; rarely will the model do something like add the backlink before the content it links to. The kinds of anomalies admitted by the protocol (in the sense defined by isolation levels) all appear benign when the updates proceed in this prefix-safe way. The intelligence is capable of determining actions to take on the retry, and that's the point of our protocol. For example, in our testing, one reasoning trace reported, "I already have this content in the compaction-service page from a previous update. Let me check—the section ‘April 2026 production incident: Node SSD failure’ is already there and has source_ids that include most of these.” From the source ids it was able to determine that it had already applied the data. Leaving work in a partially-done state requires a model smart enough to retry based upon the state it observes. In practice, Haiku 4.5 with Context-1 as a search sub-agent was sufficient for wiki generation. There are two modes of operation that can lead a transaction to be wounded: Waiting to read or waiting to write. For the former, we observed that in approximately 39.7% of wounded transactions, the model would revisit via read or write the page that wounded its prior attempt. For the latter, we observed that of pages written during a wounded transaction's retry, that 28.2% of modifications share a prefix or suffix with the page causing the wounded abort. Thus we see a non-trivial number of wounded transactions revisit the work that wounded them. Our empirical conclusion is that workers will block waiting for a lock to read a page they will never write, and this is not false sharing. An earlier prototype used reader-writer locks instead of pure exclusive locks. The rate of wounding was higher and upgrading locks was not sufficient to make progress. By this point, an astute reader will have reached for several existing familiar models that don't quite fit because: Why not store the wiki in git and access it via a filesystem? It's a natural question to ask. There are two reasons to not go with this approach: First, it requires secondary indexing in a searchable way; else, it incurs massive token costs. Second, git is terrible for resolving merge conflicts. An early version of Foundation used Git for conflict resolution atop a file-system-like abstraction. 3/8 of the calls explicitly gave up after a repeated conflict. 4/8 gave up due to their read being stale or requiring a re-read to write because someone else wrote out from under them. The remaining case (correctly) concluded that the write was unnecessary and therefore it abandoned the write. Some memorable quotes: Every concurrency control protocol embeds a trade-off that expends one thing to gain another. Textbook protocols assume that the work is cheap, redoing it is near-free, and atomicity is the thing to conserve. An AI agent's work is many minutes of reasoning and dollars of tokens. Fission trades atomicity in order to conserve reasoning, because a valid, reasoned-through edit is worth more than a clean history. Intelligence is left to reconcile the committed work with what remains to be committed. Everything else is textbook.