# FAILED is not UNKNOWN: the retry bug hiding in every AI agent > Source: > Published: 2026-09-08 17:14:58+00:00 An agent refunds a customer $500. Stripe processes it. The response never comes back — a proxy timeout, a dropped connection, a container that got OOM-killed mid-call. Your code sees an exception. Your retry decorator does what retry decorators do. Now the customer has $1,000. Nothing in that sequence is an LLM problem. The model reasoned correctly, picked the right tool, and passed the right arguments. The bug is in the four lines of infrastructure everybody writes without thinking: ``` for attempt in range(3): try: return stripe.Refund.create(payment_intent=pid, amount=amount) except Exception: time.sleep(2 ** attempt) raise ``` That code encodes an assumption that is simply false: **that an error means it didn't happen.** Almost every retry system in the wild models outcomes as a boolean. Success, or failure. Returned, or raised. A call that leaves your process has three possible outcomes: | Outcome | What you know | Safe to retry? | |---|---|---| | `COMMITTED` | The remote system acted, and you have proof | No — it's done | | `FAILED` | The remote system did not act, and you have proof | Yes | | `AMBIGUOUS` | You have no idea | **No** | `AMBIGUOUS` is not a rare edge case. It is the normal result of a timeout, a connection reset, a 502 from a load balancer, a gateway that gave up before the origin did, or your own process dying between the request and the response. In distributed systems this has a name — the two generals problem — and it has no clean solution. What it has is a discipline: never collapse "unknown" into "failed." Databases have understood this for forty years. That's what two-phase commit is about. Payment providers have understood it for twenty; that's what an idempotency key is. Agent frameworks are ten months into shipping software that takes consequential action, and most of them still have a `max_retries` parameter and no concept of an unknown outcome at all. The difference now is who's driving. A cron job retries in one predictable shape. An LLM retries because it read an error string, decided the action didn't go through, and reasoned its way to trying again — sometimes with slightly different arguments, sometimes three turns later, sometimes from a different worker. It will do this confidently, and it will tell you it succeeded. The instinct is to write a ledger: ``` result = do_refund(payment_id, amount) db.mark_done(f"refund:{payment_id}") # too late return result ``` This does nothing. The window you care about is exactly the window where you have no row: the call is in flight, the process dies, and the retry arrives to find an empty table. You have to claim the effect *before* the call: ``` key = f"refund:{payment_id}" if not store.reserve(key): # atomic insert, unique constraint raise DuplicateEffect(key) # someone already claimed this try: result = do_refund(payment_id, amount) except TimeoutError: store.mark_ambiguous(key) # held, NOT released raise except ProviderRejected: store.mark_failed(key) # provably didn't happen — safe to release raise else: store.commit(key, result) return result ``` Read the `except` blocks twice. The entire safety property lives there. A timeout does not release the reservation. That reservation stays held until something outside the agent settles it: a reconciliation call to the provider, or a human. `AMBIGUOUS` is a state you *live in*, not a state you clear by guessing. And the key itself matters. `refund:txn_4821` is the identity of a business action. It has to be the same string across a retry, a second worker, a restart, and a fresh conversation with the model. If your key includes a timestamp, a UUID, or a trace ID, you don't have deduplication — you have a log. The same class of bug shows up in human-in-the-loop flows, and it's uglier because it looks like it's working. A person approves a $500 refund. The agent gets `approved: true` back. Two turns later the agent re-plans, decides the amount should be $5,000, and calls the tool. It still holds an approval. The approval is a boolean, and booleans don't remember what they were about. An approval should be bound to the exact arguments the human read: ``` approval_hash = sha256(canonical_json({ "action": "stripe.refund", "payment_id": "txn_4821", "amount": 500_00, })).hexdigest() ``` Change one field and the hash no longer matches, so the approval authorises nothing and the call stops. Same principle for single use: an approval that can be replayed is a permission, and you didn't mean to grant a permission. While we're here — a tool being present in the agent's tool list is not permission either. "The model can call it" and "this principal may perform it, with these arguments, right now" are different questions, and only one of them is answered by your prompt. Most of the safety tooling in this space watches what the agent *says*: prompt injection filters, output classifiers, jailbreak detection, PII scrubbing. All useful, all aimed at the model. None of it helps here. The model was fine. The failure happened in the gap between "the agent decided" and "the real system changed" — one function call wide, no natural language in it at all. That gap needs a different kind of check, one that never sees a prompt and only sees an action with its exact arguments: Five questions, asked in the last moment before the effect is real. I got tired of writing the reservation table by hand on every project, so I built the thing. [CTRLRun](https://ctrlrun.dev) is an open-source Python library (Apache-2.0) that sits at that execution boundary. It's a library inside your process, not a service in front of it. It never sees your prompts, your model, or your reasoning traces. ``` pip install ctrlrun python import ctrlrun @ctrlrun.protect( "stripe.refund", effect="refund:{payment_id}", ) def refund(payment_id, amount): ... ``` What you get around that call: `allow` → the effect is reserved, then your function runs`approve` → `ApprovalRequired` is raised, bound to these exact arguments`deny` → `ActionDenied`, and no approval request is even created` DuplicateEffect`, with the original receipt returned` AMBIGUOUS`, and the blind retry gets `AmbiguousEffect` SQLite on one host, Postgres across hosts. State survives a restart, which is the point — a process that died mid-call still leaves a claim behind for the retry to hit. There's an MCP gateway if your agent talks over MCP, and a `ctrlrun verify` command that checks a set of guarantees in CI. There's a browser demo at [ctrlrun.dev](https://ctrlrun.dev) that walks through each of these failures across a bunch of domains — no signup, and the "try it" page runs the actual released wheel in your tab via Pyodide, so the refusals you see are the library's own. If you only keep one thing from this: go find the retry logic wrapped around whatever your agents do to production, and check what it does with a timeout. If it retries, you have this bug. It hasn't cost you anything yet because your volume is low and most timeouts really are failures. Most of them. Source: [github.com/CTRLRun/ctrlrun](https://github.com/CTRLRun/ctrlrun). Issues and disagreement both welcome — particularly if you've hit a failure mode I haven't modelled.