{"slug": "tool-design-for-reliable-multi-agent-crews-preventing-coordination-failures-at", "title": "Tool design for reliable multi-agent crews: preventing coordination failures at the tool boundary", "summary": "A developer writing on the Loop & Retry blog argues that multi-agent coordination failures such as split-brain writes, goal drift and context drift should be prevented at the tool boundary rather than through prompt-level discipline. The post shows that a naive read_state/write_state API makes stale writes the only expressible operation, and proposes compare-and-set-style signatures with required version parameters so that a stale write becomes unrepresentable and is rejected at call time with a structured, retryable error.", "body_md": "*Originally published on [Loop & Retry](https://loopandretry.github.io/posts/multi-agent-tool-design-for-coordination/?ref=devto) — field notes on building LLM agents that survive production.*\n\nThis cluster has spent four posts on how a crew of agents breaks. [Split-brain](https://loopandretry.github.io/posts/multi-agent-memory-split-brain/?ref=devto) is two agents writing to a shared store from different baselines. [Goal drift](https://loopandretry.github.io/posts/multi-agent-goal-drift/?ref=devto) is the objective decaying one handoff at a time. [Context drift](https://loopandretry.github.io/posts/multi-agent-context-drift/?ref=devto) is the crew's beliefs about a fixed objective going stale unevenly. [Observability for multi-agent systems](https://loopandretry.github.io/posts/multi-agent-observability/?ref=devto) is how you see any of it happening, on top of the four modes in [failure modes in multi-agent teams](https://loopandretry.github.io/posts/multi-agent-failure-modes/?ref=devto).\n\nRead the remedies at the end of those posts and a pattern shows up. \"Give writes an expected version.\" \"Carry constraints verbatim at every hop.\" \"Re-pull the objective at checkpoints instead of trusting the copy you started with.\" Every one of those is a *discipline* — a thing some agent has to remember to do, correctly, on every call, forever. That's a bad bet. This post is about moving the same rules somewhere they don't depend on anyone remembering them.\n\nAn agent re-derives its behavior from context on every call. Tell it \"always pass the version you read\" in a system prompt and you have a strong hint, not a guarantee — it competes with everything else in the window, and it degrades exactly when the run gets long and complicated, which is when coordination failures happen. A tool signature is different in kind. It's the one artifact in the loop the model cannot reason its way around, because the validation runs in your code after the call is made. A required parameter is not advice.\n\nSo the design goal is sharper than \"write good tools.\" It's to make the failure **unrepresentable**: a tool that has nowhere to put a stale write cannot be used to make one. You are not detecting the coordination failure or reminding an agent to avoid it. You are deleting the state space where it lives.\n\nThat's also the cheapest fix available, on the [economics of catching things late](https://loopandretry.github.io/posts/cost-of-late-failure-detection/?ref=devto). P9's instrumentation is worth building and it is fundamentally *retrospective* — a divergent-reads query tells you two agents disagreed, after they did. A compare-and-set signature means the second write was rejected at the moment it was attempted, with a structured error the caller can act on. One produces an incident review; the other produces a retry.\n\n[Designing tools an LLM won't misuse](https://loopandretry.github.io/posts/designing-tools-an-llm-wont-misuse/?ref=devto) laid out four properties that make a tool hard to fumble: a legible schema, a validating boundary, recoverable errors, and idempotency. All four assume one caller. Coordination failures need a fifth: **the tool has to know it isn't the only caller.** Every signature below is that assumption made explicit.\n\nHere's the shape that causes it, and it's the default shape of nearly every shared-memory wrapper I've seen:\n\n```\n# BAD: there is nowhere to put \"the version I read\", so every write is a blind write\ndef read_state(key: str) -> str: ...\ndef write_state(key: str, value: str) -> None: ...\n```\n\nThe bug isn't that an agent *can* write from a stale baseline. It's that a stale write is the only kind of write this API can express. `write_state` accepts no evidence that the caller ever read the key, so the store cannot distinguish \"I checked and I'm updating from current\" from \"I'm clobbering whatever's there.\" Nothing an agent does at the call site can fix that; the information is absent from the signature.\n\n``` python\n# GOOD: reads hand out a version, writes must present one\ndef read_state(key: str) -> dict:\n    \"\"\"Read shared state. Returns {\"value\": str, \"version\": str}.\n    The version token must be passed to write_state to commit a change\n    based on this value.\"\"\"\n\ndef write_state(key: str, value: str, expected_version: str) -> dict:\n    \"\"\"Commit a change to shared state. expected_version must be the version\n    token from the read this value was derived from. If the key has changed\n    since that read, the write is rejected and the current value is returned —\n    re-read, redo the work against it, and retry.\"\"\"\n```\n\nThree details carry most of the weight.\n\n**`expected_version` has no default.** A default — `None`, `-1`, `\"latest\"` — reintroduces the entire failure mode, because the model will supply it whenever it doesn't have a version handy, which is precisely the case you were guarding against. Required means required.\n\n**The version is an opaque token, not an integer.** Give a model an `int` version and it will occasionally reason about it: pass `expected_version=5` because it read 4 and is about to make one change. Arithmetic on a version defeats the check. An opaque string is a token to carry, not a number to compute.\n\n**The rejection returns the current value.** This is [error-as-prompt](https://loopandretry.github.io/posts/tool-output-is-untrusted-input/?ref=devto) applied to concurrency: a bare `ConflictError` gives the model nothing to do but retry identically and fail identically. Returning `{\"rejected\": true, \"current_value\": ..., \"current_version\": ...}` lets it redo the work against the real baseline in one more turn.\n\nIf your shared store is exposed over MCP, this is exactly the composition problem [MCP tool-design mistakes](https://loopandretry.github.io/posts/mcp-server-tool-design-mistakes/?ref=devto) describes — a server built for one client, then plugged into a session with four agents whose existence its author never considered.\n\nGoal drift enters through the handoff, and the handoff is usually a tool call. This one:\n\n```\n# BAD: \"context\" is a free-text field, which makes it a paraphrase channel\ndef delegate(subtask: str, context: str) -> dict: ...\n```\n\nA free-text `context` parameter invites exactly the behavior that causes drift: the delegating agent summarizes what it thinks matters. Summarizing *is* paraphrasing, paraphrasing loses boundary conditions first — they aren't tasks, so they don't survive a rewrite aimed at tasks — and the loss is invisible because the resulting handoff reads perfectly well.\n\n```\n# GOOD: constraints are structured, required, and verbatim; dropping one is an event\ndef delegate(\n    subtask: str,\n    origin_task_id: str,\n    constraints: list[dict],          # [{\"id\": str, \"text\": str}] — text copied,\n                                      # never rewritten, from the origin ledger\n    acceptance_criteria: list[dict],  # [{\"check\": str, \"constraint_id\": str}]\n    waived_constraints: list[dict],   # [{\"id\": str, \"reason\": str}]\n) -> dict:\n    \"\"\"Delegate a subtask. Every constraint id on the origin task must appear in\n    either constraints (carried forward verbatim) or waived_constraints (with a\n    reason). A constraint that appears in neither is a validation error, not a\n    silent omission. Each acceptance criterion must cite the constraint id it\n    checks; criteria citing no constraint are rejected.\"\"\"\n```\n\nThe mechanism is a completeness check the tool performs, not the agent. It knows `origin_task_id`, so it can look up that task's constraint ledger and verify the partition: carried, or waived with a reason. There is no third option, so *forgetting* stops being possible. Dropping a constraint is still allowed — sometimes it genuinely doesn't apply downstream — but it now costs a written reason and leaves a record naming the hop that did it. That converts P9's \"constraint C3 was last seen at hop 2\" from a forensic query into a rejected call.\n\nRequiring each acceptance criterion to cite a constraint id closes the other half. The `?cache=bypass` change in the goal-drift post passed review because the builder's definition of done never mentioned the API contract. A criterion that can't name what it traces to is one nobody is checking against the original ask, and the schema can simply refuse it.\n\nContext drift is a freshness failure, so freshness belongs in the response envelope — not in a doc telling agents to re-read things.\n\n```\n# BAD: the objective arrives as a string with no freshness metadata at all\ndef get_objective(task_id: str) -> str: ...\n```\n\nA bare string is indistinguishable from a fresh one no matter how old it is. That's the [context window as a cache](https://loopandretry.github.io/posts/context-window-is-a-cache/?ref=devto) with no expiry field.\n\n```\n# GOOD: every read carries its own age and whether it has been superseded\ndef get_objective(task_id: str, agent_id: str) -> dict:\n    \"\"\"Returns {\"objective\": str, \"version\": str, \"as_of\": iso8601,\n    \"superseded\": bool, \"refresh_required\": bool}. Registers agent_id as a\n    holder of this version, so later corrections can identify it as stale.\"\"\"\n```\n\nRegistering the reader on read is what makes correction fan-out computable at all — the holder set builds itself as a side effect of normal use, rather than depending on someone maintaining a subscriber list.\n\nThen make the *acting* tool care:\n\n```\n# GOOD: the tool that touches the outside world refuses a superseded basis\ndef send_customer_message(ticket_id: str, body: str, policy_version: str) -> dict:\n    \"\"\"Send a customer-facing message. policy_version must be the version from\n    the get_objective read this message was drafted against. If that version has\n    been superseded, the send is rejected with the current policy — re-read and\n    redraft.\"\"\"\n```\n\nThat signature is the $32-credit bug from the context-drift post, closed. The responder in that story did everything right against the policy it held; it just held a superseded copy, and nothing between its context and the customer knew or cared. Putting `policy_version` on the send makes the outbound side of the crew the enforcement point — which is the right place, per P9's blast-radius argument, because the last tool in the chain has nothing after it to catch a mistake.\n\n**The shared scratchpad.** `append_note(text: str)` plus `read_notes() -> str`. Comfortable, ubiquitous, and structurally the worst option available: no keys, so nothing to group by; no versions, so no conflict is detectable; no writer identity, so no diff is attributable. It manufactures split-brain and simultaneously removes every field that would let you observe it — the missing join key from P9, at the source.\n\n**The unversioned get/set pair.** Covered above, and worth naming separately because it's what you get by default from wrapping any KV store, cache, or document API. The wrapper is where the version has to be introduced; if the underlying store has one, don't hide it.\n\n**The tool that silently merges.** `update_state(key, partial: dict)` that deep-merges and returns `{\"ok\": true}` is the most dangerous of the three, because it looks like the sophisticated choice. Two agents writing disjoint fields succeed and produce a record neither one intended, resolved by arrival order, reported as success twice. If a merge can't tell you how many collisions it resolved, it isn't logging the thing that matters — and a merge that *can* tell you should be returning the conflict, not absorbing it.\n\n**The `force: bool` escape hatch.** Any of the above signatures plus `force=True` is the unversioned original with extra steps. A model that has just been rejected by a concurrency check and can see a `force` parameter in the schema will find it. Don't ship the bypass in the same tool as the guard; make forcing a separate, differently-named, human-triggered operation.\n\nCoordination failures happen at boundaries between agents, and in a working system every one of those boundaries is a tool call. That's the leverage: you don't have to make N agents disciplined, you have to make one schema strict. Put a version token on every read and require it on every write, and split-brain becomes a rejected call. Make constraints a required structured parameter of the handoff and waivers explicit, and goal drift becomes a validation error naming the hop. Put age and supersession in the response envelope and require a policy version on the tools that touch the outside world, and context drift becomes a refused send instead of a promise to a customer nobody is operating on.\n\nNone of this replaces the instrumentation — you still want the counters, because a rising rejection rate is how you learn contention is growing before it hurts, and rejected writes only show up as a metric if you're counting them. But the ordering matters. Instrument what you can't prevent; prevent what you can. If your crew's shared-state tool has a `write` that takes no version, no dashboard you build on top of it will be as valuable as the twenty minutes it takes to add the parameter.\n\nOnce the signatures are in place, prove they hold: [testing multi-agent crews](https://loopandretry.github.io/posts/testing-multi-agent-coordination-failures/?ref=devto) shows how to force each of these failures deterministically in CI, with no model in the loop.", "url": "https://wpnews.pro/news/tool-design-for-reliable-multi-agent-crews-preventing-coordination-failures-at", "canonical_source": "https://dev.to/loopandretry/tool-design-for-reliable-multi-agent-crews-preventing-coordination-failures-at-the-tool-boundary-2hbj", "published_at": "2026-10-23 16:05:45+00:00", "updated_at": "2026-10-23 16:12:37.504399+00:00", "lang": "en", "topics": ["ai-agents", "ai-tools", "developer-tools", "artificial-intelligence"], "entities": ["Loop & Retry"], "also_reported_by": [], "alternates": {"html": "https://wpnews.pro/news/tool-design-for-reliable-multi-agent-crews-preventing-coordination-failures-at", "markdown": "https://wpnews.pro/news/tool-design-for-reliable-multi-agent-crews-preventing-coordination-failures-at.md", "text": "https://wpnews.pro/news/tool-design-for-reliable-multi-agent-crews-preventing-coordination-failures-at.txt", "jsonld": "https://wpnews.pro/news/tool-design-for-reliable-multi-agent-crews-preventing-coordination-failures-at.jsonld"}}