Tool design for reliable multi-agent crews: preventing coordination failures at the tool boundary

A developer writing on the Loop & Retry blog argues that multi-agent coordination failures such as split-brain writes, goal drift and context drift should be prevented at the tool boundary rather than through prompt-level discipline. The post shows that a naive read_state/write_state API makes stale writes the only expressible operation, and proposes compare-and-set-style signatures with required version parameters so that a stale write becomes unrepresentable and is rejected at call time with a structured, retryable error.

Originally published on Loop & Retry https://loopandretry.github.io/posts/multi-agent-tool-design-for-coordination/?ref=devto — field notes on building LLM agents that survive production. This cluster has spent four posts on how a crew of agents breaks. Split-brain https://loopandretry.github.io/posts/multi-agent-memory-split-brain/?ref=devto is two agents writing to a shared store from different baselines. Goal drift https://loopandretry.github.io/posts/multi-agent-goal-drift/?ref=devto is the objective decaying one handoff at a time. Context drift https://loopandretry.github.io/posts/multi-agent-context-drift/?ref=devto is the crew's beliefs about a fixed objective going stale unevenly. Observability for multi-agent systems https://loopandretry.github.io/posts/multi-agent-observability/?ref=devto is how you see any of it happening, on top of the four modes in failure modes in multi-agent teams https://loopandretry.github.io/posts/multi-agent-failure-modes/?ref=devto . Read the remedies at the end of those posts and a pattern shows up. "Give writes an expected version." "Carry constraints verbatim at every hop." "Re-pull the objective at checkpoints instead of trusting the copy you started with." Every one of those is a discipline — a thing some agent has to remember to do, correctly, on every call, forever. That's a bad bet. This post is about moving the same rules somewhere they don't depend on anyone remembering them. An agent re-derives its behavior from context on every call. Tell it "always pass the version you read" in a system prompt and you have a strong hint, not a guarantee — it competes with everything else in the window, and it degrades exactly when the run gets long and complicated, which is when coordination failures happen. A tool signature is different in kind. It's the one artifact in the loop the model cannot reason its way around, because the validation runs in your code after the call is made. A required parameter is not advice. So the design goal is sharper than "write good tools." It's to make the failure unrepresentable : a tool that has nowhere to put a stale write cannot be used to make one. You are not detecting the coordination failure or reminding an agent to avoid it. You are deleting the state space where it lives. That's also the cheapest fix available, on the economics of catching things late https://loopandretry.github.io/posts/cost-of-late-failure-detection/?ref=devto . P9's instrumentation is worth building and it is fundamentally retrospective — a divergent-reads query tells you two agents disagreed, after they did. A compare-and-set signature means the second write was rejected at the moment it was attempted, with a structured error the caller can act on. One produces an incident review; the other produces a retry. Designing tools an LLM won't misuse https://loopandretry.github.io/posts/designing-tools-an-llm-wont-misuse/?ref=devto laid out four properties that make a tool hard to fumble: a legible schema, a validating boundary, recoverable errors, and idempotency. All four assume one caller. Coordination failures need a fifth: the tool has to know it isn't the only caller. Every signature below is that assumption made explicit. Here's the shape that causes it, and it's the default shape of nearly every shared-memory wrapper I've seen: BAD: there is nowhere to put "the version I read", so every write is a blind write def read state key: str - str: ... def write state key: str, value: str - None: ... The bug isn't that an agent can write from a stale baseline. It's that a stale write is the only kind of write this API can express. write state accepts no evidence that the caller ever read the key, so the store cannot distinguish "I checked and I'm updating from current" from "I'm clobbering whatever's there." Nothing an agent does at the call site can fix that; the information is absent from the signature. python GOOD: reads hand out a version, writes must present one def read state key: str - dict: """Read shared state. Returns {"value": str, "version": str}. The version token must be passed to write state to commit a change based on this value.""" def write state key: str, value: str, expected version: str - dict: """Commit a change to shared state. expected version must be the version token from the read this value was derived from. If the key has changed since that read, the write is rejected and the current value is returned — re-read, redo the work against it, and retry.""" Three details carry most of the weight. expected version has no default. A default — None , -1 , "latest" — reintroduces the entire failure mode, because the model will supply it whenever it doesn't have a version handy, which is precisely the case you were guarding against. Required means required. The version is an opaque token, not an integer. Give a model an int version and it will occasionally reason about it: pass expected version=5 because it read 4 and is about to make one change. Arithmetic on a version defeats the check. An opaque string is a token to carry, not a number to compute. The rejection returns the current value. This is error-as-prompt https://loopandretry.github.io/posts/tool-output-is-untrusted-input/?ref=devto applied to concurrency: a bare ConflictError gives the model nothing to do but retry identically and fail identically. Returning {"rejected": true, "current value": ..., "current version": ...} lets it redo the work against the real baseline in one more turn. If your shared store is exposed over MCP, this is exactly the composition problem MCP tool-design mistakes https://loopandretry.github.io/posts/mcp-server-tool-design-mistakes/?ref=devto describes — a server built for one client, then plugged into a session with four agents whose existence its author never considered. Goal drift enters through the handoff, and the handoff is usually a tool call. This one: BAD: "context" is a free-text field, which makes it a paraphrase channel def delegate subtask: str, context: str - dict: ... A free-text context parameter invites exactly the behavior that causes drift: the delegating agent summarizes what it thinks matters. Summarizing is paraphrasing, paraphrasing loses boundary conditions first — they aren't tasks, so they don't survive a rewrite aimed at tasks — and the loss is invisible because the resulting handoff reads perfectly well. GOOD: constraints are structured, required, and verbatim; dropping one is an event def delegate subtask: str, origin task id: str, constraints: list dict , {"id": str, "text": str} — text copied, never rewritten, from the origin ledger acceptance criteria: list dict , {"check": str, "constraint id": str} waived constraints: list dict , {"id": str, "reason": str} - dict: """Delegate a subtask. Every constraint id on the origin task must appear in either constraints carried forward verbatim or waived constraints with a reason . A constraint that appears in neither is a validation error, not a silent omission. Each acceptance criterion must cite the constraint id it checks; criteria citing no constraint are rejected.""" The mechanism is a completeness check the tool performs, not the agent. It knows origin task id , so it can look up that task's constraint ledger and verify the partition: carried, or waived with a reason. There is no third option, so forgetting stops being possible. Dropping a constraint is still allowed — sometimes it genuinely doesn't apply downstream — but it now costs a written reason and leaves a record naming the hop that did it. That converts P9's "constraint C3 was last seen at hop 2" from a forensic query into a rejected call. Requiring each acceptance criterion to cite a constraint id closes the other half. The ?cache=bypass change in the goal-drift post passed review because the builder's definition of done never mentioned the API contract. A criterion that can't name what it traces to is one nobody is checking against the original ask, and the schema can simply refuse it. Context drift is a freshness failure, so freshness belongs in the response envelope — not in a doc telling agents to re-read things. BAD: the objective arrives as a string with no freshness metadata at all def get objective task id: str - str: ... A bare string is indistinguishable from a fresh one no matter how old it is. That's the context window as a cache https://loopandretry.github.io/posts/context-window-is-a-cache/?ref=devto with no expiry field. GOOD: every read carries its own age and whether it has been superseded def get objective task id: str, agent id: str - dict: """Returns {"objective": str, "version": str, "as of": iso8601, "superseded": bool, "refresh required": bool}. Registers agent id as a holder of this version, so later corrections can identify it as stale.""" Registering the reader on read is what makes correction fan-out computable at all — the holder set builds itself as a side effect of normal use, rather than depending on someone maintaining a subscriber list. Then make the acting tool care: GOOD: the tool that touches the outside world refuses a superseded basis def send customer message ticket id: str, body: str, policy version: str - dict: """Send a customer-facing message. policy version must be the version from the get objective read this message was drafted against. If that version has been superseded, the send is rejected with the current policy — re-read and redraft.""" That signature is the $32-credit bug from the context-drift post, closed. The responder in that story did everything right against the policy it held; it just held a superseded copy, and nothing between its context and the customer knew or cared. Putting policy version on the send makes the outbound side of the crew the enforcement point — which is the right place, per P9's blast-radius argument, because the last tool in the chain has nothing after it to catch a mistake. The shared scratchpad. append note text: str plus read notes - str . Comfortable, ubiquitous, and structurally the worst option available: no keys, so nothing to group by; no versions, so no conflict is detectable; no writer identity, so no diff is attributable. It manufactures split-brain and simultaneously removes every field that would let you observe it — the missing join key from P9, at the source. The unversioned get/set pair. Covered above, and worth naming separately because it's what you get by default from wrapping any KV store, cache, or document API. The wrapper is where the version has to be introduced; if the underlying store has one, don't hide it. The tool that silently merges. update state key, partial: dict that deep-merges and returns {"ok": true} is the most dangerous of the three, because it looks like the sophisticated choice. Two agents writing disjoint fields succeed and produce a record neither one intended, resolved by arrival order, reported as success twice. If a merge can't tell you how many collisions it resolved, it isn't logging the thing that matters — and a merge that can tell you should be returning the conflict, not absorbing it. The force: bool escape hatch. Any of the above signatures plus force=True is the unversioned original with extra steps. A model that has just been rejected by a concurrency check and can see a force parameter in the schema will find it. Don't ship the bypass in the same tool as the guard; make forcing a separate, differently-named, human-triggered operation. Coordination failures happen at boundaries between agents, and in a working system every one of those boundaries is a tool call. That's the leverage: you don't have to make N agents disciplined, you have to make one schema strict. Put a version token on every read and require it on every write, and split-brain becomes a rejected call. Make constraints a required structured parameter of the handoff and waivers explicit, and goal drift becomes a validation error naming the hop. Put age and supersession in the response envelope and require a policy version on the tools that touch the outside world, and context drift becomes a refused send instead of a promise to a customer nobody is operating on. None of this replaces the instrumentation — you still want the counters, because a rising rejection rate is how you learn contention is growing before it hurts, and rejected writes only show up as a metric if you're counting them. But the ordering matters. Instrument what you can't prevent; prevent what you can. If your crew's shared-state tool has a write that takes no version, no dashboard you build on top of it will be as valuable as the twenty minutes it takes to add the parameter. Once the signatures are in place, prove they hold: testing multi-agent crews https://loopandretry.github.io/posts/testing-multi-agent-coordination-failures/?ref=devto shows how to force each of these failures deterministically in CI, with no model in the loop.