{"slug": "your-ai-agent-charged-the-customer-twice-and-every-dashboard-stayed-green", "title": "Your AI Agent Charged the Customer Twice and Every Dashboard Stayed Green", "summary": "A developer detailed a silent failure class in LLM agents with write access, where a timeout triggers stacked retries across the HTTP client, orchestration framework, and model itself, causing the same irreversible action — such as a payment charge — to execute twice while every monitoring dashboard stays green. The proposed fix is a DedupGate that fingerprints each action as a hash of action name, sorted parameters, and caller scope, blocks repeated signatures inside a sliding 300-second window, and exposes a Duplicate Action Rate (DAR) metric, with a minimal Python implementation provided.", "body_md": "An AI agent hit a timeout. It retried the payment API call. The first request had already succeeded.\n\nEvery monitoring dashboard stayed green.\n\nThis is a silent failure class that traditional SRE metrics miss completely. We ask \"did the request return HTTP 200?\" The question that matters for agents with write access is:\n\nDid the agent execute the same irreversible action twice?\n\nWhen an LLM agent operates under non-deterministic retries, status codes stop telling you the truth about what happened.\n\nWhy agents make this worse\n\nNormal retry logic is boring and predictable. Agents are not:\n\nRetries stack. The HTTP client retries, the orchestration framework retries, and the model decides to \"try again\". Each layer assumes it is the only one.\n\nRetries get rephrased. After a timeout the agent may re-plan and re-issue the action with reordered or slightly different parameters, so no lower layer recognizes it as a duplicate.\n\nA timeout means \"unknown\", not \"failed\". The agent can't tell if the write landed, and it tends to guess \"do it again.\"\n\nThree rules for critical write paths\n\nFingerprint the action before dispatching it:\n\nkey = hash(action_name + sorted_parameters + caller_scope)\n\nIf the payload matches, the agent cannot emit a duplicate mutation. Sort keys and normalize types so logically identical payloads hash the same. Include the caller scope (tenant, session, workflow run) so unrelated users don't collide, and an intent ID if the same action can legitimately repeat.\n\nBlock repeated signatures inside a bounded window (say, 300 seconds).\n\nDon't swallow duplicates silently. That hides the bug behind them. Intercept the call, reject it with an explicit error, and increment a counter. \"Sliding\" means each blocked attempt refreshes the window, so an agent stuck in a loop stays blocked.\n\nIf DAR spikes, don't blame the agent. It is almost always compensating for unhandled timeouts, slow database locks or upstream lag. DAR tends to show you those problems before your customers do.\n\nA minimal gate in Python\n\npython\n\nimport hashlib\n\nimport json\n\nimport threading\n\nimport time\n\nclass DuplicateActionError(Exception):\n\n    \"\"\"Raised when an identical action was already submitted inside the window.\"\"\"\n\nclass DedupGate:\n\n    def **init**(self, ttl_seconds: float = 300, clock=time.monotonic):\n\n        self.ttl = ttl_seconds\n\n        self._clock = clock\n\n        self._seen: dict[str, float] = {}\n\n        self._lock = threading.Lock()\n\n        self.allowed = 0\n\n        self.blocked = 0\n\n``` python\n@staticmethod\ndef action_key(action_name: str, params: dict, caller_scope: str) -> str:\n    canonical = json.dumps(\n        {\"a\": action_name, \"p\": params, \"s\": caller_scope},\n        sort_keys=True, separators=(\",\", \":\"), default=str,\n    )\n    return hashlib.sha256(canonical.encode()).hexdigest()\n\ndef acquire(self, action_name: str, params: dict, caller_scope: str) -> str:\n    key = self.action_key(action_name, params, caller_scope)\n    now = self._clock()\n    with self._lock:\n        # evict expired entries\n        self._seen = {k: t for k, t in self._seen.items() if now - t < self.ttl}\n        if key in self._seen:\n            self._seen[key] = now  # sliding: refresh the window\n            self.blocked += 1\n            raise DuplicateActionError(\n                f\"'{action_name}' already submitted within {self.ttl:.0f}s\"\n            )\n        self._seen[key] = now\n        self.allowed += 1\n        return key\n\ndef release(self, key: str) -> None:\n    \"\"\"Call only on a *confirmed* failure, so a legitimate retry can proceed.\"\"\"\n    with self._lock:\n        self._seen.pop(key, None)\n\n@property\ndef dar(self) -> float:\n    total = self.allowed + self.blocked\n    return self.blocked / total if total else 0.0\n```\n\nUsing it around a payment call:\n\npython\n\ngate = DedupGate(ttl_seconds=300)\n\ndef charge(order_id: str, amount_cents: int, tenant: str):\n\n    params = {\"order_id\": order_id, \"amount_cents\": amount_cents}\n\n    key = gate.acquire(\"create_charge\", params, caller_scope=tenant)\n\n    try:\n\n        # also pass `key` as the provider's idempotency key if it supports one\n\n        return payments.create_charge(**params, idempotency_key=key)\n\n    except PaymentDeclined:\n\n        gate.release(key)   # definitive failure: safe to retry\n\n        raise\n\n    # on a timeout: do NOT release. Reconcile by looking up the charge instead.\n\nThe agent gets a typed error it can reason about (\"already submitted, check status\") instead of looping.\n\nGotchas\n\nAmbiguous timeouts. Don't release the key on a timeout. Reconcile against the downstream system first. Release only on definitive failure.\n\nIn-memory state doesn't scale out. This gate lives in one process. With multiple workers, back it with a shared atomic store, for example Redis SET key value NX EX 300, or a DynamoDB conditional write.\n\nDefense in depth. Forward the action key as the provider's idempotency key where available. The gate protects you from your agent, and the provider key protects you from everything else.\n\nLegitimate repeats. Two deliberate $10 transfers are not duplicates. Put an intent ID into the key.\n\nWhere this lives\n\nI integrated this deduplication gate into the open-source agentsre telemetry framework so DAR sits next to the other signals an agent emits: [https://lnkd.in/e7Qpux2j](https://lnkd.in/e7Qpux2j)\n\nDiscussion\n\nHow is your team handling idempotency and duplicate execution as agents move from read-only assistants to production write systems? Enforced at the agent layer, the API layer, or both? I'd like to hear the war stories.", "url": "https://wpnews.pro/news/your-ai-agent-charged-the-customer-twice-and-every-dashboard-stayed-green", "canonical_source": "https://dev.to/ajaydevineni/your-ai-agent-charged-the-customer-twice-and-every-dashboard-stayed-green-269p", "published_at": "2026-10-17 01:04:06+00:00", "updated_at": "2026-10-17 01:27:05.589507+00:00", "lang": "en", "topics": ["ai-agents", "mlops", "ai-safety", "ai-tools"], "entities": ["DedupGate", "DuplicateActionError"], "also_reported_by": [], "alternates": {"html": "https://wpnews.pro/news/your-ai-agent-charged-the-customer-twice-and-every-dashboard-stayed-green", "markdown": "https://wpnews.pro/news/your-ai-agent-charged-the-customer-twice-and-every-dashboard-stayed-green.md", "text": "https://wpnews.pro/news/your-ai-agent-charged-the-customer-twice-and-every-dashboard-stayed-green.txt", "jsonld": "https://wpnews.pro/news/your-ai-agent-charged-the-customer-twice-and-every-dashboard-stayed-green.jsonld"}}