{"slug": "the-chatbot-refunded-twice-the-retry-looked-successful", "title": "The Chatbot Refunded Twice. The Retry Looked Successful", "summary": "A developer documented a failure pattern in AI agents that perform consequential actions through APIs, using a composite example in which a chatbot issued a duplicate $89 refund after a payment request timed out. The retry appeared successful because the provider processed the same business action twice, and the agent could not distinguish a lost response from a genuine failure. The proposed fix assigns a stable actionId generated from trusted workflow state, passes it as an idempotency key, enforces a unique database constraint, and replaces a simple SUCCESS/FAILURE/RETRY model with states including CONFIRMATION_UNKNOWN and REQUIRES_REVIEW.", "body_md": "The production trace looked harmless:\n\n```\n14:03:11  Refund requested for order 4821\n14:03:12  Payment provider called\n14:03:22  Request timed out\n14:03:24  Agent selected retry\n14:03:26  Refund completed: $89\n```\n\nThe chatbot reported success. The customer received confirmation. No exception remained open.\n\nTwo days later, finance found two $89 refunds.\n\nThe first payment request had not failed. Only the response had failed to reach the chatbot. When the agent retried, the payment provider processed the same business action again.\n\nThe retry looked successful because it was successful.\n\nThat was the bug.\n\nThis is a simplified composite example of a failure pattern that appears whenever an AI agent can perform consequential actions through APIs.\n\nDevelopers often treat a timeout as evidence that an operation failed.\n\nIt is only evidence that the client did not receive a response within the expected time.\n\nSeveral things may have happened:\n\nThe chatbot cannot distinguish these outcomes from a timeout alone.\n\nA human support agent might pause, check the payment dashboard, and confirm the order status. An AI agent may interpret the timeout as permission to repeat the tool call.\n\nThis is why “retry on failure” is not a complete reliability policy.\n\nSuppose the agent calls this tool:\n\n```\nrefundPayment({\n  orderId: \"4821\",\n  amount: 89\n});\n```\n\nAfter a timeout, the model generates the same arguments and calls the tool again. The two calls look identical to us, but the payment system may see two independent refund requests.\n\nThe application should create a stable action identifier before execution:\n\n```\ntype RefundCommand = {\n  actionId: string;\n  orderId: string;\n  amount: number;\n  requestedBy: string;\n};\n\nconst command: RefundCommand = {\n  actionId: \"refund:4821:approved-request-7\",\n  orderId: \"4821\",\n  amount: 89,\n  requestedBy: \"support-agent\"\n};\n```\n\nThe important field is `actionId`.\n\nIt represents the business action, not one HTTP attempt. Every retry of that refund must reuse the same identifier.\n\nDo not ask the language model to invent or remember this value. The surrounding application should generate it from trusted workflow state and persist it before calling the payment provider.\n\nIf the provider supports idempotency keys, pass the stable identifier with every attempt:\n\n```\nasync function issueRefund(command: RefundCommand) {\n  return paymentProvider.refund(\n    {\n      orderId: command.orderId,\n      amount: command.amount\n    },\n    {\n      idempotencyKey: command.actionId\n    }\n  );\n}\n```\n\nThe first request may complete without returning a response. When the retry arrives with the same key, the provider should return the original result instead of creating another refund.\n\nThat protects the external transaction, but the application still needs its own record:\n\n```\nawait db.refundCommands.create({\n  data: {\n    actionId: command.actionId,\n    orderId: command.orderId,\n    amount: command.amount,\n    status: \"PENDING\"\n  }\n});\n```\n\nPlace a unique constraint on `actionId`.\n\nIf two workers receive the same command simultaneously, only one should be allowed to create the execution record. The other should read the existing state instead of starting another action.\n\nIf an external API does not support idempotency, a database check alone may not be sufficient. Two processes can both check before either records completion. Use a lock, serialized queue, transactional outbox, or another mechanism that guarantees one executor for that business action.\n\nThe chatbot’s original workflow understood three outcomes:\n\n```\nSUCCESS\nFAILURE\nRETRY\n```\n\nThat was too simple.\n\nThe refund command needed states that described what the system actually knew:\n\n```\nPENDING\nEXECUTING\nCONFIRMATION_UNKNOWN\nCOMPLETED\nREJECTED\nREQUIRES_REVIEW\n```\n\nA timeout after sending the request should move the command to `CONFIRMATION_UNKNOWN`, not directly to `FAILURE`.\n\nFrom there, the system can query the provider using the idempotency key, inspect a webhook, or send the case to a human. The chatbot should not independently decide that repeating a financial action is safe.\n\nMost tests simulate an API rejecting a request before doing any work.\n\nThe more dangerous case is a successful operation followed by a lost response.\n\nA useful integration test should reproduce this sequence:\n\nThen run the test with two workers executing the same command concurrently.\n\nThe assertion that matters is not how many HTTP attempts occurred. It is how many financial transactions were created.\n\nA 2026 [Kore.ai survey](https://www.kore.ai/research/agent-productivity-index-report) of 408 enterprise IT and engineering leaders found that 79% of consequential autonomous actions required manual reversal. Among those reversals, 93% were described as costly or disruptive.\n\nBetter prompts cannot solve a missing transaction boundary.\n\nIf you are evaluating an [AI chatbot development company](https://spaculus.com/ai-chatbot-development-company/), do not ask only whether its chatbot can process a refund. Ask what happens when the refund succeeds but its response disappears.\n\nA reliable chatbot must know more than how to complete an action.\n\nIt must know when the action may already be complete.", "url": "https://wpnews.pro/news/the-chatbot-refunded-twice-the-retry-looked-successful", "canonical_source": "https://dev.to/paul-s/the-chatbot-refunded-twice-the-retry-looked-successful-cgk", "published_at": "2026-10-20 11:40:56+00:00", "updated_at": "2026-10-20 11:48:32.820356+00:00", "lang": "en", "topics": ["ai-agents", "ai-safety", "ai-tools"], "entities": [], "also_reported_by": [], "alternates": {"html": "https://wpnews.pro/news/the-chatbot-refunded-twice-the-retry-looked-successful", "markdown": "https://wpnews.pro/news/the-chatbot-refunded-twice-the-retry-looked-successful.md", "text": "https://wpnews.pro/news/the-chatbot-refunded-twice-the-retry-looked-successful.txt", "jsonld": "https://wpnews.pro/news/the-chatbot-refunded-twice-the-retry-looked-successful.jsonld"}}