When an AI agent's HTTP request or browser tool call times out, what does your system record?
If it records failed
, the agent has a blind spot. A network timeout does not mean the operation failed on the remote server; it means the connection closed before the client received the answer. If the server processed the mutation, retrying the call blindly will create a duplicate artifact: a double payment, a duplicate ticket, a repeat email, or a redundant article.
If it records succeeded
, it is hallucinating certainty.
The missing state is ** outcome_unknown**—a first-class operational status that halts automatic retries, records the unconfirmed mutation, and hands off execution to an explicit reconciliation loop.
In our previous post, we discussed why agents need action receipts rather than purely semantic memory. Following valuable discussions with practitioners on distributed systems and memory boundaries, this article turns that concept into a concrete, testable state machine you can drop into any production agent framework.
Not all exceptions are created equal:
[Intent Recorded]
|
v
[Attempting Transport] ---> (DNS / Local Socket / Auth error) ---> [REJECTED / SAFE_TO_RETRY]
|
(Bytes sent)
|
v
[Awaiting Response] ---> (Connection Timeout / Drop / 504) ---> [OUTCOME_UNKNOWN]
Treating post-send ambiguity as a failure is the root cause of automated duplicate storms.
Here is the complete lifecycle of a guarded agent action:
| State | Type | Description | Allowed Next Transitions |
|---|---|---|---|
planned |
|||
| Transient | Intent recorded locally with safe payload fingerprint. | ||
submitted , rejected |
|||
submitted |
|||
| Transient | Bytes sent to remote endpoint; awaiting response. | ||
succeeded , rejected , outcome_unknown |
|||
outcome_unknown |
|||
| Suspended | Network dropped or timed out after submission. Retries blocked. | ||
reconciling , manual_review |
|||
reconciling |
|||
| Active | Querying external system for proof of effect. | ||
succeeded , safe_to_retry , manual_review |
|||
succeeded |
|||
| Terminal | External ID verified via response or readback. | None | |
safe_to_retry |
|||
| Terminal | Absence of effect proven via authoritative readback. | None (new intent required) | |
rejected |
|||
| Terminal | Server returned deterministic client error (4xx). | None | |
manual_review |
|||
| Terminal | Absence/presence cannot be proven programmatically. | Human intervention |
In payment engineering, distributed consensus is achieved through at-least-once delivery paired with a server-side deduplication key (an Idempotency Key).
When an external platform natively supports idempotency headers (such as Idempotency-Key: <uuid>
in Stripe or GitHub GraphQL mutation keys), reconciliation is straightforward: if you time out, you re-send with the exact same key.
However, the vast majority of web APIs, CRUD services, and browser-driven surfaces do not support native idempotency keys. In those environments, the caller must carry the burden:
outcome_unknown
triggers, the agent queries the read API (or search endpoint) for resources created by the agent's account within a bounded timestamp window matching the intent fingerprint.
import hashlib
import json
def compute_intent_fingerprint(method: str, path: str, payload: dict) -> str:
canonical = json.dumps(
{"method": method.upper(), "path": path, "payload": payload},
sort_keys=True,
separators=(",", ":")
)
return f"sha256:{hashlib.sha256(canonical.encode('utf-8')).hexdigest()}"
A subtle trap in state machine design is destructive in-place updates.
If an action goes submitted -> outcome_unknown -> reconciling -> succeeded
, and you simply overwrite the state to succeeded
, you destroy the historical record that the action lived in an indeterminate state for hours.
During post-mortems (or when auditing race conditions where another worker observed missing state during that window), knowing how an action reached success is as critical as the final state.
A robust action receipt preserves the full transition trajectory:
{
"operation_id": "20260818T190000Z-a1b2c3d4e5",
"operation": "articles.create",
"state": "succeeded",
"intent_fingerprint": "sha256:4d8a...",
"state_history": [
{ "state": "planned", "recorded_at": "2026-08-18T19:00:00Z" },
{ "state": "submitted", "recorded_at": "2026-08-18T19:00:01Z" },
{
"state": "outcome_unknown",
"recorded_at": "2026-08-18T19:00:31Z",
"error": { "code": "timeout", "message": "Gateway Timeout 504" }
},
{
"state": "reconciling",
"recorded_at": "2026-08-18T19:05:00Z"
},
{
"state": "succeeded",
"recorded_at": "2026-08-18T19:05:02Z",
"reconciliation": {
"evidence": "Readback from /api/articles matched title fingerprint",
"external_id": 4407310
}
}
]
}
Here is a Python implementation of the guarded execution and reconciliation pattern:
from dataclasses import dataclass, field
from datetime import datetime, timezone
import uuid
@dataclass
class ActionReceipt:
operation_id: str
action: str
target: str
fingerprint: str
state: str = "planned"
external_id: str | None = None
state_history: list[dict] = field(default_factory=list)
def transition_to(self, new_state: str, **meta):
self.state = new_state
self.state_history.append({
"state": new_state,
"recorded_at": datetime.now(timezone.utc).isoformat(),
**meta
})
def execute_guarded_action(client, action: str, target: str, payload: dict) -> ActionReceipt:
receipt = ActionReceipt(
operation_id=f"{datetime.now(timezone.utc).strftime('%Y%m%dT%H%M%SZ')}-{uuid.uuid4().hex[:8]}",
action=action,
target=target,
fingerprint=compute_intent_fingerprint("POST", target, payload)
)
receipt.transition_to("planned")
persist_receipt(receipt)
receipt.transition_to("submitted")
persist_receipt(receipt)
try:
response = client.post(target, json=payload, timeout=10.0)
receipt.external_id = response.json().get("id")
receipt.transition_to("succeeded", status_code=response.status_code)
except TimeoutError as exc:
receipt.transition_to("outcome_unknown", error=str(exc))
except Exception as exc:
receipt.transition_to("rejected", error=str(exc))
finally:
persist_receipt(receipt)
return receipt
def reconcile_receipt(client, receipt: ActionReceipt, read_fn) -> ActionReceipt:
if receipt.state != "outcome_unknown":
return receipt
receipt.transition_to("reconciling")
persist_receipt(receipt)
matched_item = read_fn(client, receipt.fingerprint)
if matched_item:
receipt.external_id = matched_item["id"]
receipt.transition_to("succeeded", evidence="Matched on readback query")
else:
receipt.transition_to("safe_to_retry", evidence="Authoritative readback showed 0 records")
persist_receipt(receipt)
return receipt
manual_review
.DELETE
) are inherently trickier to reconcile because absence is the intended target state. A missing item could mean either the delete succeeded or the item never existed.When building autonomous agents interacting with external APIs or browser surfaces:
Which external write in your systems is hardest to reconcile after an unexpected timeout, and how do you prevent duplicate execution?