An AI agent hit a timeout. It retried the payment API call. The first request had already succeeded.
Every monitoring dashboard stayed green.
This is a silent failure class that traditional SRE metrics miss completely. We ask "did the request return HTTP 200?" The question that matters for agents with write access is:
Did the agent execute the same irreversible action twice?
When an LLM agent operates under non-deterministic retries, status codes stop telling you the truth about what happened.
Why agents make this worse
Normal retry logic is boring and predictable. Agents are not:
Retries stack. The HTTP client retries, the orchestration framework retries, and the model decides to "try again". Each layer assumes it is the only one.
Retries get rephrased. After a timeout the agent may re-plan and re-issue the action with reordered or slightly different parameters, so no lower layer recognizes it as a duplicate.
A timeout means "unknown", not "failed". The agent can't tell if the write landed, and it tends to guess "do it again."
Three rules for critical write paths
Fingerprint the action before dispatching it:
key = hash(action_name + sorted_parameters + caller_scope)
If the payload matches, the agent cannot emit a duplicate mutation. Sort keys and normalize types so logically identical payloads hash the same. Include the caller scope (tenant, session, workflow run) so unrelated users don't collide, and an intent ID if the same action can legitimately repeat.
Block repeated signatures inside a bounded window (say, 300 seconds).
Don't swallow duplicates silently. That hides the bug behind them. Intercept the call, reject it with an explicit error, and increment a counter. "Sliding" means each blocked attempt refreshes the window, so an agent stuck in a loop stays blocked.
If DAR spikes, don't blame the agent. It is almost always compensating for unhandled timeouts, slow database locks or upstream lag. DAR tends to show you those problems before your customers do.
A minimal gate in Python
python
import hashlib
import json
import threading
import time
class DuplicateActionError(Exception):
"""Raised when an identical action was already submitted inside the window."""
class DedupGate:
def **init**(self, ttl_seconds: float = 300, clock=time.monotonic):
self.ttl = ttl_seconds
self._clock = clock
self._seen: dict[str, float] = {}
self._lock = threading.Lock()
self.allowed = 0
self.blocked = 0
@staticmethod
def action_key(action_name: str, params: dict, caller_scope: str) -> str:
canonical = json.dumps(
{"a": action_name, "p": params, "s": caller_scope},
sort_keys=True, separators=(",", ":"), default=str,
)
return hashlib.sha256(canonical.encode()).hexdigest()
def acquire(self, action_name: str, params: dict, caller_scope: str) -> str:
key = self.action_key(action_name, params, caller_scope)
now = self._clock()
with self._lock:
self._seen = {k: t for k, t in self._seen.items() if now - t < self.ttl}
if key in self._seen:
self._seen[key] = now # sliding: refresh the window
self.blocked += 1
raise DuplicateActionError(
f"'{action_name}' already submitted within {self.ttl:.0f}s"
)
self._seen[key] = now
self.allowed += 1
return key
def release(self, key: str) -> None:
"""Call only on a *confirmed* failure, so a legitimate retry can proceed."""
with self._lock:
self._seen.pop(key, None)
@property
def dar(self) -> float:
total = self.allowed + self.blocked
return self.blocked / total if total else 0.0
Using it around a payment call:
python
gate = DedupGate(ttl_seconds=300)
def charge(order_id: str, amount_cents: int, tenant: str):
params = {"order_id": order_id, "amount_cents": amount_cents}
key = gate.acquire("create_charge", params, caller_scope=tenant)
try:
return payments.create_charge(**params, idempotency_key=key)
except PaymentDeclined:
gate.release(key) # definitive failure: safe to retry
raise
The agent gets a typed error it can reason about ("already submitted, check status") instead of looping.
Gotchas
Ambiguous timeouts. Don't release the key on a timeout. Reconcile against the downstream system first. Release only on definitive failure.
In-memory state doesn't scale out. This gate lives in one process. With multiple workers, back it with a shared atomic store, for example Redis SET key value NX EX 300, or a DynamoDB conditional write.
Defense in depth. Forward the action key as the provider's idempotency key where available. The gate protects you from your agent, and the provider key protects you from everything else.
Legitimate repeats. Two deliberate $10 transfers are not duplicates. Put an intent ID into the key.
Where this lives
I integrated this deduplication gate into the open-source agentsre telemetry framework so DAR sits next to the other signals an agent emits: https://lnkd.in/e7Qpux2j
Discussion
How is your team handling idempotency and duplicate execution as agents move from read-only assistants to production write systems? Enforced at the agent layer, the API layer, or both? I'd like to hear the war stories.