A timeout is not a failed action. It is an unknown action.
That distinction matters the first time an agent calls a payment API, sends a customer email, or creates a CRM record—and loses the response. If the agent retries blindly, it may double-charge a card or send the same message twice. If it does nothing, the user’s request may never finish. A prompt telling the model to “avoid duplicates” cannot settle that ambiguity.
The fix is idempotency: make each logical write happen once, even when workers crash, queues redeliver, SDKs retry, or the model asks to try again. This guide shows a practical design for putting that guarantee in the tool layer, where it can be tested and enforced.
Every distributed system has ambiguous failures. A request may reach the target, commit its side effect, and then lose its response. Traditional backends solve this with idempotency keys, unique constraints, durable jobs, and reconciliation.
Agents amplify the problem because they are built to keep trying. A model sees timeout, changes its plan, and asks the same tool to run again. Meanwhile, a queue may redeliver the job after a lease expires, and an HTTP client may retry underneath both of them. One user intent can become four writes.
Treat these as distinct outcomes:
| Outcome | What the caller knows | Safe next step |
|---|---|---|
| Rejected before send | The target did not receive it | Correct input, then retry if appropriate |
| Explicit failure | The target responded with a permanent error | Stop or request correction |
| Explicit success | The target returned a durable receipt | Store and return the receipt |
| Unknown commit state | The request may have succeeded | Reconcile before retrying |
The last row is the important one. A timeout after send must never be silently mapped to “failed.”
An idempotency key identifies the business action, not a network attempt.
For example, “send the approved invoice email for invoice inv_123 revision 4” is one action. It should keep the same key if a worker restarts five times. A later resend after a user edits the invoice is a new action and needs a new key.
Good keys are stable, scoped, and inspectable:
tenant:acme | run:run_8f2 | step:email_invoice | invoice:inv_123 | revision:4
Hash that canonical representation if it contains sensitive identifiers. Do not generate a new UUID on every retry; that turns a dedupe mechanism into a duplicate generator. Also do not use only the user prompt. “Email this invoice” can be a valid request more than once.
An action should usually include:
The agent should request a business action, not manipulate retry behavior directly. Place a deterministic action runner between the model and every state-changing integration.
agent plan
-> tool request (business intent)
-> action runner (policy + operation ledger)
-> external API / database
-> receipt + reconciliation result
-> agent
This boundary is useful even if your tools are ordinary functions. It gives the system one place to validate tenant access, lock the operation, pass provider keys, classify failures, and hide unsafe retry choices from the model.
Reads can often be retried. Writes need an operation record first.
Postgres is enough for many teams. Create the operation before calling the external service; keep it until the action’s replay window is over.
create table agent_operations (
id uuid primary key,
tenant_id uuid not null,
idempotency_key text not null,
action_type text not null,
args_hash text not null,
status text not null check (status in (
'pending', 'running', 'succeeded', 'unknown', 'failed', 'needs_review'
)),
provider_reference text,
result_json jsonb,
error_json jsonb,
lease_expires_at timestamptz,
created_at timestamptz not null default now(),
updated_at timestamptz not null default now(),
unique (tenant_id, idempotency_key)
);
The unique constraint is a hard concurrency boundary. Two workers can receive the same job, but only one can own the logical action. Store an args_hash too: if the same key arrives with different arguments, fail closed. Reusing a key with changed data is almost always a caller bug.
pending means no worker has started the side effect. running means a worker holds a short lease. succeeded stores the exact receipt to return on a duplicate call. unknown means the downstream result is ambiguous, so a reconciliation job—not an agent—must decide what happened.
Avoid marking an operation failed simply because the HTTP request timed out. That destroys the evidence needed to prevent a duplicate.
Here is a simplified pattern. In production, the claim query should use a transaction and a lease so a crashed worker can be recovered safely.
type ActionResult = { receiptId: string; status: "sent" | "already_sent" };
async function sendInvoiceEmail(input: {
tenantId: string;
runId: string;
invoiceId: string;
revision: number;
to: string;
}): Promise<ActionResult> {
const key = `invoice-email:${input.tenantId}:${input.invoiceId}:${input.revision}`;
const argsHash = sha256(JSON.stringify({ to: input.to, revision: input.revision }));
const operation = await operations.claim({
tenantId: input.tenantId,
key,
actionType: "invoice_email",
argsHash,
leaseSeconds: 60,
});
if (operation.status === "succeeded") return operation.resultJson;
if (operation.status === "unknown") return reconcileInvoiceEmail(operation, input);
if (operation.status !== "running") throw new Error("Action is not safe to execute");
try {
const response = await emailProvider.send({
to: input.to,
template: "invoice",
metadata: { operationId: operation.id },
idempotencyKey: key,
});
return await operations.succeed(operation.id, {
receiptId: response.messageId,
status: "sent",
});
} catch (error) {
if (isAmbiguousTransportError(error)) {
await operations.markUnknown(operation.id, serialize(error));
return reconcileInvoiceEmail(operation, input);
}
await operations.fail(operation.id, serialize(error));
throw error;
}
}
Notice what is missing: no model instruction decides whether an unknown write gets replayed. The wrapper returns a receipt, a verified “already completed” result, or an explicit escalation.
Many third-party APIs do not support idempotency keys—or claim to accept them without making them searchable. Your internal ledger still helps, but it cannot prove the external effect occurred.
Make the action observable at the target. Depending on the system, reconciliation can query:
For an email, attach operationId as provider metadata and persist the provider message ID. For a CRM create, send an external ID derived from the operation key. For an internal database mutation, use a unique operation_id column or an INSERT ... ON CONFLICT pattern.
If reconciliation cannot prove success or failure, keep the operation in needs_review. This is safer than inventing an answer. The agent can tell the user that the action is pending verification rather than claiming it completed.
“Retry three times” is too blunt. A safe policy distinguishes failure modes.
| Failure class | Example | Retry policy |
|---|---|---|
| Safe read | Search endpoint returns 503 | Exponential backoff with a limit |
| Validation error | Missing required field | Do not retry; return correction needed |
| Rate limit | HTTP 429 | Wait for the provider signal, then retry same operation |
| Ambiguous write | Socket closes after request body | Mark unknown and reconcile |
| Known transient write failure | Provider confirms no commit | Retry same operation key |
| Auth or policy denial | Scope removed | Stop and escalate |
Retry budgets should also be outside the model. Put limits on attempts, elapsed time, spend, and allowed action types. This makes a bad downstream day finite instead of turning it into an overnight retry loop.
Unit tests that expect a 200 response are not enough. Build a small fault-injection suite around every write tool.
Test at least these cases:
succeeded is stored.
An effective assertion is simple: after any number of retries, the target contains exactly one effect for the operation key. Also test the user-visible result: a duplicate request should return the original receipt, not an opaque error.
Track these counters by tenant, tool, and provider:
unknown
These are more useful than raw timeout counts. A rise in unknown commits may reveal a provider regression, an overly short client timeout, or a worker shutdown problem. A rise in duplicate calls can mean your queue is redelivering as designed—or that an upstream client is misbehaving.
Add an audit event each time an operation moves state. Include the actor, agent run, tool version, key hash, arguments hash, provider reference, and decision made by reconciliation. Do not store sensitive prompt or customer data just for convenience.
Start with the highest-impact tools: payments, email sends, record creation, access changes, and anything that triggers work outside your system. Inventory every write-side tool and give it an action type, stable key recipe, reconciliation strategy, and owner.
Then ship in stages:
This is a better investment than ever more prompt rules. Prompts can help an agent choose a valid action; they cannot offer exactly-once delivery across a network.
AI agent idempotency means repeating the same logical agent action produces one durable effect, not multiple ones. It protects state-changing tools from retries, worker crashes, duplicate queue messages, and repeated model tool calls.
Every state-changing tool should have an idempotency strategy. Read-only tools can normally use conventional retry policies. For writes, use provider keys, internal unique constraints, or an operation ledger plus reconciliation.
No. A prompt influences model behavior but cannot coordinate concurrent workers, recover a lost response, or prove whether a remote API committed an action. Enforce deduplication in the action runner and target system.
Treat it as an unknown commit state. Record the ambiguity, query the system of record using a correlation ID or business key, and retry only if you can establish that no effect occurred.
Keep them at least as long as every possible replay window: queue retention, client retries, scheduled job retries, and provider webhook delays. High-risk actions such as payments often need longer retention and durable audit evidence.
No. Most infrastructure offers at-least-once delivery. Idempotency makes repeated delivery safe by ensuring the receiver commits a logical action once and returns the original result to later attempts.
Reliable agent systems do not pretend a timeout means failure. They preserve the operation, reconcile the target, and only then decide whether a retry is safe. Give every write tool a stable identity and a durable receipt, and an agent can be persistent without becoming destructive.