Originally published on Loop & Retry — field notes on building LLM agents that survive production.
This cluster has spent four posts on how a crew of agents breaks. Split-brain is two agents writing to a shared store from different baselines. Goal drift is the objective decaying one handoff at a time. Context drift is the crew's beliefs about a fixed objective going stale unevenly. Observability for multi-agent systems is how you see any of it happening, on top of the four modes in failure modes in multi-agent teams.
Read the remedies at the end of those posts and a pattern shows up. "Give writes an expected version." "Carry constraints verbatim at every hop." "Re-pull the objective at checkpoints instead of trusting the copy you started with." Every one of those is a discipline — a thing some agent has to remember to do, correctly, on every call, forever. That's a bad bet. This post is about moving the same rules somewhere they don't depend on anyone remembering them.
An agent re-derives its behavior from context on every call. Tell it "always pass the version you read" in a system prompt and you have a strong hint, not a guarantee — it competes with everything else in the window, and it degrades exactly when the run gets long and complicated, which is when coordination failures happen. A tool signature is different in kind. It's the one artifact in the loop the model cannot reason its way around, because the validation runs in your code after the call is made. A required parameter is not advice.
So the design goal is sharper than "write good tools." It's to make the failure unrepresentable: a tool that has nowhere to put a stale write cannot be used to make one. You are not detecting the coordination failure or reminding an agent to avoid it. You are deleting the state space where it lives.
That's also the cheapest fix available, on the economics of catching things late. P9's instrumentation is worth building and it is fundamentally retrospective — a divergent-reads query tells you two agents disagreed, after they did. A compare-and-set signature means the second write was rejected at the moment it was attempted, with a structured error the caller can act on. One produces an incident review; the other produces a retry.
Designing tools an LLM won't misuse laid out four properties that make a tool hard to fumble: a legible schema, a validating boundary, recoverable errors, and idempotency. All four assume one caller. Coordination failures need a fifth: the tool has to know it isn't the only caller. Every signature below is that assumption made explicit.
Here's the shape that causes it, and it's the default shape of nearly every shared-memory wrapper I've seen:
def read_state(key: str) -> str: ...
def write_state(key: str, value: str) -> None: ...
The bug isn't that an agent can write from a stale baseline. It's that a stale write is the only kind of write this API can express. write_state accepts no evidence that the caller ever read the key, so the store cannot distinguish "I checked and I'm updating from current" from "I'm clobbering whatever's there." Nothing an agent does at the call site can fix that; the information is absent from the signature.
def read_state(key: str) -> dict:
"""Read shared state. Returns {"value": str, "version": str}.
The version token must be passed to write_state to commit a change
based on this value."""
def write_state(key: str, value: str, expected_version: str) -> dict:
"""Commit a change to shared state. expected_version must be the version
token from the read this value was derived from. If the key has changed
since that read, the write is rejected and the current value is returned —
re-read, redo the work against it, and retry."""
Three details carry most of the weight.
expected_version has no default. A default — None, -1, "latest" — reintroduces the entire failure mode, because the model will supply it whenever it doesn't have a version handy, which is precisely the case you were guarding against. Required means required.
The version is an opaque token, not an integer. Give a model an int version and it will occasionally reason about it: pass expected_version=5 because it read 4 and is about to make one change. Arithmetic on a version defeats the check. An opaque string is a token to carry, not a number to compute.
The rejection returns the current value. This is error-as-prompt applied to concurrency: a bare ConflictError gives the model nothing to do but retry identically and fail identically. Returning {"rejected": true, "current_value": ..., "current_version": ...} lets it redo the work against the real baseline in one more turn.
If your shared store is exposed over MCP, this is exactly the composition problem MCP tool-design mistakes describes — a server built for one client, then plugged into a session with four agents whose existence its author never considered.
Goal drift enters through the handoff, and the handoff is usually a tool call. This one:
def delegate(subtask: str, context: str) -> dict: ...
A free-text context parameter invites exactly the behavior that causes drift: the delegating agent summarizes what it thinks matters. Summarizing is paraphrasing, paraphrasing loses boundary conditions first — they aren't tasks, so they don't survive a rewrite aimed at tasks — and the loss is invisible because the resulting handoff reads perfectly well.
def delegate(
subtask: str,
origin_task_id: str,
constraints: list[dict], # [{"id": str, "text": str}] — text copied,
acceptance_criteria: list[dict], # [{"check": str, "constraint_id": str}]
waived_constraints: list[dict], # [{"id": str, "reason": str}]
) -> dict:
"""Delegate a subtask. Every constraint id on the origin task must appear in
either constraints (carried forward verbatim) or waived_constraints (with a
reason). A constraint that appears in neither is a validation error, not a
silent omission. Each acceptance criterion must cite the constraint id it
checks; criteria citing no constraint are rejected."""
The mechanism is a completeness check the tool performs, not the agent. It knows origin_task_id, so it can look up that task's constraint ledger and verify the partition: carried, or waived with a reason. There is no third option, so forgetting stops being possible. Dropping a constraint is still allowed — sometimes it genuinely doesn't apply downstream — but it now costs a written reason and leaves a record naming the hop that did it. That converts P9's "constraint C3 was last seen at hop 2" from a forensic query into a rejected call.
Requiring each acceptance criterion to cite a constraint id closes the other half. The ?cache=bypass change in the goal-drift post passed review because the builder's definition of done never mentioned the API contract. A criterion that can't name what it traces to is one nobody is checking against the original ask, and the schema can simply refuse it.
Context drift is a freshness failure, so freshness belongs in the response envelope — not in a doc telling agents to re-read things.
def get_objective(task_id: str) -> str: ...
A bare string is indistinguishable from a fresh one no matter how old it is. That's the context window as a cache with no expiry field.
def get_objective(task_id: str, agent_id: str) -> dict:
"""Returns {"objective": str, "version": str, "as_of": iso8601,
"superseded": bool, "refresh_required": bool}. Registers agent_id as a
holder of this version, so later corrections can identify it as stale."""
Registering the reader on read is what makes correction fan-out computable at all — the holder set builds itself as a side effect of normal use, rather than depending on someone maintaining a subscriber list.
Then make the acting tool care:
def send_customer_message(ticket_id: str, body: str, policy_version: str) -> dict:
"""Send a customer-facing message. policy_version must be the version from
the get_objective read this message was drafted against. If that version has
been superseded, the send is rejected with the current policy — re-read and
redraft."""
That signature is the $32-credit bug from the context-drift post, closed. The responder in that story did everything right against the policy it held; it just held a superseded copy, and nothing between its context and the customer knew or cared. Putting policy_version on the send makes the outbound side of the crew the enforcement point — which is the right place, per P9's blast-radius argument, because the last tool in the chain has nothing after it to catch a mistake.
The shared scratchpad. append_note(text: str) plus read_notes() -> str. Comfortable, ubiquitous, and structurally the worst option available: no keys, so nothing to group by; no versions, so no conflict is detectable; no writer identity, so no diff is attributable. It manufactures split-brain and simultaneously removes every field that would let you observe it — the missing join key from P9, at the source.
The unversioned get/set pair. Covered above, and worth naming separately because it's what you get by default from wrapping any KV store, cache, or document API. The wrapper is where the version has to be introduced; if the underlying store has one, don't hide it.
The tool that silently merges. update_state(key, partial: dict) that deep-merges and returns {"ok": true} is the most dangerous of the three, because it looks like the sophisticated choice. Two agents writing disjoint fields succeed and produce a record neither one intended, resolved by arrival order, reported as success twice. If a merge can't tell you how many collisions it resolved, it isn't logging the thing that matters — and a merge that can tell you should be returning the conflict, not absorbing it.
The force: bool escape hatch. Any of the above signatures plus force=True is the unversioned original with extra steps. A model that has just been rejected by a concurrency check and can see a force parameter in the schema will find it. Don't ship the bypass in the same tool as the guard; make forcing a separate, differently-named, human-triggered operation.
Coordination failures happen at boundaries between agents, and in a working system every one of those boundaries is a tool call. That's the leverage: you don't have to make N agents disciplined, you have to make one schema strict. Put a version token on every read and require it on every write, and split-brain becomes a rejected call. Make constraints a required structured parameter of the handoff and waivers explicit, and goal drift becomes a validation error naming the hop. Put age and supersession in the response envelope and require a policy version on the tools that touch the outside world, and context drift becomes a refused send instead of a promise to a customer nobody is operating on.
None of this replaces the instrumentation — you still want the counters, because a rising rejection rate is how you learn contention is growing before it hurts, and rejected writes only show up as a metric if you're counting them. But the ordering matters. Instrument what you can't prevent; prevent what you can. If your crew's shared-state tool has a write that takes no version, no dashboard you build on top of it will be as valuable as the twenty minutes it takes to add the parameter.
Once the signatures are in place, prove they hold: testing multi-agent crews shows how to force each of these failures deterministically in CI, with no model in the loop.