Your AI Agent Charged the Customer Twice and Every Dashboard Stayed Green

A developer detailed a silent failure class in LLM agents with write access, where a timeout triggers stacked retries across the HTTP client, orchestration framework, and model itself, causing the same irreversible action — such as a payment charge — to execute twice while every monitoring dashboard stays green. The proposed fix is a DedupGate that fingerprints each action as a hash of action name, sorted parameters, and caller scope, blocks repeated signatures inside a sliding 300-second window, and exposes a Duplicate Action Rate (DAR) metric, with a minimal Python implementation provided.

An AI agent hit a timeout. It retried the payment API call. The first request had already succeeded. Every monitoring dashboard stayed green. This is a silent failure class that traditional SRE metrics miss completely. We ask "did the request return HTTP 200?" The question that matters for agents with write access is: Did the agent execute the same irreversible action twice? When an LLM agent operates under non-deterministic retries, status codes stop telling you the truth about what happened. Why agents make this worse Normal retry logic is boring and predictable. Agents are not: Retries stack. The HTTP client retries, the orchestration framework retries, and the model decides to "try again". Each layer assumes it is the only one. Retries get rephrased. After a timeout the agent may re-plan and re-issue the action with reordered or slightly different parameters, so no lower layer recognizes it as a duplicate. A timeout means "unknown", not "failed". The agent can't tell if the write landed, and it tends to guess "do it again." Three rules for critical write paths Fingerprint the action before dispatching it: key = hash action name + sorted parameters + caller scope If the payload matches, the agent cannot emit a duplicate mutation. Sort keys and normalize types so logically identical payloads hash the same. Include the caller scope tenant, session, workflow run so unrelated users don't collide, and an intent ID if the same action can legitimately repeat. Block repeated signatures inside a bounded window say, 300 seconds . Don't swallow duplicates silently. That hides the bug behind them. Intercept the call, reject it with an explicit error, and increment a counter. "Sliding" means each blocked attempt refreshes the window, so an agent stuck in a loop stays blocked. If DAR spikes, don't blame the agent. It is almost always compensating for unhandled timeouts, slow database locks or upstream lag. DAR tends to show you those problems before your customers do. A minimal gate in Python python import hashlib import json import threading import time class DuplicateActionError Exception : """Raised when an identical action was already submitted inside the window.""" class DedupGate: def init self, ttl seconds: float = 300, clock=time.monotonic : self.ttl = ttl seconds self. clock = clock self. seen: dict str, float = {} self. lock = threading.Lock self.allowed = 0 self.blocked = 0 python @staticmethod def action key action name: str, params: dict, caller scope: str - str: canonical = json.dumps {"a": action name, "p": params, "s": caller scope}, sort keys=True, separators= ",", ":" , default=str, return hashlib.sha256 canonical.encode .hexdigest def acquire self, action name: str, params: dict, caller scope: str - str: key = self.action key action name, params, caller scope now = self. clock with self. lock: evict expired entries self. seen = {k: t for k, t in self. seen.items if now - t < self.ttl} if key in self. seen: self. seen key = now sliding: refresh the window self.blocked += 1 raise DuplicateActionError f"'{action name}' already submitted within {self.ttl:.0f}s" self. seen key = now self.allowed += 1 return key def release self, key: str - None: """Call only on a confirmed failure, so a legitimate retry can proceed.""" with self. lock: self. seen.pop key, None @property def dar self - float: total = self.allowed + self.blocked return self.blocked / total if total else 0.0 Using it around a payment call: python gate = DedupGate ttl seconds=300 def charge order id: str, amount cents: int, tenant: str : params = {"order id": order id, "amount cents": amount cents} key = gate.acquire "create charge", params, caller scope=tenant try: also pass key as the provider's idempotency key if it supports one return payments.create charge params, idempotency key=key except PaymentDeclined: gate.release key definitive failure: safe to retry raise on a timeout: do NOT release. Reconcile by looking up the charge instead. The agent gets a typed error it can reason about "already submitted, check status" instead of looping. Gotchas Ambiguous timeouts. Don't release the key on a timeout. Reconcile against the downstream system first. Release only on definitive failure. In-memory state doesn't scale out. This gate lives in one process. With multiple workers, back it with a shared atomic store, for example Redis SET key value NX EX 300, or a DynamoDB conditional write. Defense in depth. Forward the action key as the provider's idempotency key where available. The gate protects you from your agent, and the provider key protects you from everything else. Legitimate repeats. Two deliberate $10 transfers are not duplicates. Put an intent ID into the key. Where this lives I integrated this deduplication gate into the open-source agentsre telemetry framework so DAR sits next to the other signals an agent emits: https://lnkd.in/e7Qpux2j https://lnkd.in/e7Qpux2j Discussion How is your team handling idempotency and duplicate execution as agents move from read-only assistants to production write systems? Enforced at the agent layer, the API layer, or both? I'd like to hear the war stories.