Idempotency keys: how to make a retried POST do its job once

A developer outlined a four-branch server-side implementation of the IETF draft Idempotency-Key HTTP header, which lets retried POST requests execute only once by having clients send a stable key per logical operation and servers store and replay the original response. The approach requires an atomic key claim to prevent duplicate execution across instances, returns 409 Conflict with Retry-After while an original request is still in flight, and rejects same-key/different-body retries with 422 after comparing a canonicalized body hash. The developer noted that AI agents retry more aggressively than human-written clients, making idempotency handling more pressing for APIs they call.

The client did nothing wrong, which is the part that took me a while to accept. It sent a request, got no answer, and sent it again, which is exactly what every HTTP client is told to do. The problem is that "no answer" covers two situations that look identical from the client's side: either the request never arrived, or it arrived, did its work, and the response got lost on the way back. The client can't tell which. So it retries, which is correct, and the server does the work twice, which is not. Timeouts and flaky networks have always caused this. What's newer is AI agents calling your API, and in my experience they retry harder than any human-written client, because a model that sees an error will usually just try again with the same arguments until something works. An operation is idempotent if doing it once and doing it five times leave the system in the same state and give the client the same answer. "Set the user's email to X" is idempotent. "Add 5 to the balance" is not. HTTP sorts its methods this way already. GET, HEAD, PUT and DELETE are defined as idempotent. Deleting something already gone returns 404 rather than 204, so the response changes between calls, but the state doesn't, and the state is what the definition is about. POST has no such guarantee, and can't really, because POST means "process this" and the server decides what processing means. Create an order, send an email, charge a card. The awkward thing is that the operations that actually matter in an API, the ones involving money or messages, are nearly all POSTs, so the protocol gives you nothing where you need it most. The fix has been in payment APIs for years and is now an IETF draft, "The Idempotency-Key HTTP Header Field," from the HTTPAPI working group. It's still a draft and hasn't become an RFC, but the design is simple enough that it's the convention regardless. The client generates a unique key, usually a UUID, for each logical operation. "Charge this customer $5 for this order" gets one key, and every retry of that operation sends the same key in an Idempotency-Key header. The server uses it to recognise a request it has already seen and return the original answer instead of doing the work again. The word that matters there is "logical." The key identifies what the user is trying to accomplish, not the individual HTTP request. I've seen clients generate a fresh key inside the retry loop, so every attempt carries a different one, and at that point you've implemented the header without any of the behaviour. Each unsafe request with a key lands in one of four states, and the implementation is basically these four branches. GET and HEAD skip the whole thing; they're already idempotent and storing their responses would just be a cache with worse semantics. First arrival: claim the key, run the handler, store the response status and body. The claim has to be atomic, because two copies can arrive within the same millisecond on two instances, and if the claim is a read followed by a write, both read "nothing here" and both run. Retry after the first finished: return the stored response, run nothing. I add an Idempotent-Replayed: true header so I can tell real responses from replays in logs. Retry while the first is still running: there's nothing to replay yet. Return 409 Conflict with Retry-After and let the client ask again in a second. This feels like it should be rare and it isn't, for reasons I'll get to in testing. Same key, different body: something is wrong on the client. Hash the body on the first request, store the hash, compare on every retry. A mismatch gets 422 Unprocessable Content, which is what the draft recommends. Hash a canonical form of the body, with object keys sorted, or a client that re-serialises {"a":1,"b":2} as {"b":2,"a":1} on retry gets a 422 it didn't earn. Keys expire. 24 hours is the usual window, long enough for any realistic retry schedule and short enough that the store doesn't grow forever. The middleware depends on a small interface over Redis rather than the ioredis client directly, so tests can swap in a fake. It also makes it obvious the whole thing needs three commands. python import { createHash } from "node:crypto"; import type { Request, Response, NextFunction } from "express"; declare global { namespace Express { interface Request { userId?: string; // set by auth middleware idempotencyKey?: string; // set here, for handlers to forward downstream } } } // The subset of ioredis we use. Lets tests pass a fake. export interface KeyStore { set key: string, value: string, ...args: string | number : Promise<"OK" | null ; get key: string : Promise<string | null ; del key: string : Promise<number ; } type IdempotencyRecord = | { state: "running"; hash: string } | { state: "done"; hash: string; status: number; body: unknown }; const SAFE METHODS = new Set "GET", "HEAD", "OPTIONS" ; const RUNNING TTL SEC = 30; // lock held while the handler runs; exceed it and a retry runs again const DONE TTL SEC = 60 60 24; // replay window, 24h // Sort object keys so {"a":1,"b":2} and {"b":2,"a":1} hash the same. export function canonicalJson value: unknown : string { if Array.isArray value return ${value.map canonicalJson .join "," } ; if value && typeof value === "object" { const entries = Object.keys value .sort .map k = ${JSON.stringify k }:${canonicalJson value as Record<string, unknown k } ; return {${entries.join "," }} ; } return JSON.stringify value ?? null ; } export function hashBody body: unknown : string { return createHash "sha256" .update canonicalJson body .digest "hex" ; } The claim is one Redis command. SET ... NX only writes if the key doesn't exist and returns null otherwise, so the first request wins and every other copy sees null . That one flag does all the concurrency work here, which I find slightly unnerving but is also why it's reliable. export function idempotency store: KeyStore { return async function middleware req: Request, res: Response, next: NextFunction { const key = req.header "Idempotency-Key" ; if key || SAFE METHODS.has req.method return next ; if req.userId return res.status 401 .end ; req.idempotencyKey = key; const storeKey = idem:${req.userId}:${key} ; const hash = hashBody req.body ; // Atomic claim. Only one concurrent request wins. const claimed = await store.set storeKey, JSON.stringify { state: "running", hash } , "NX", "EX", RUNNING TTL SEC, ; If the claim failed, read what's there and pick one of the three replies. js if claimed === null { const existing = JSON.parse await store.get storeKey ?? "null" as IdempotencyRecord | null; if existing return res.status 409 .set "Retry-After", "1" .end ; // lock expired between SET and GET if existing.hash == hash { return res.status 422 .json { error: "Idempotency-Key reused with a different request body" } ; } if existing.state === "running" { return res.status 409 .set "Retry-After", "1" .json { error: "Request with this key is still in progress" } ; } return res.status existing.status .set "Idempotent-Replayed", "true" .json existing.body ; } If the claim succeeded, wrap res.json so the response gets stored when the handler sends it. Anything that ends the response without storing a record, which covers thrown errors, 5xx, and handlers that use res.send , releases the claim so the client can retry. A 4xx is stored and replayed on purpose: the same body will fail validation the same way, so there's no reason to run it again to find out. // Store the response when the handler sends it. 5xx is not stored, so the client may retry it. let stored = false; const originalJson = res.json.bind res ; res.json = body: unknown = { if res.statusCode < 500 { stored = true; const record: IdempotencyRecord = { state: "done", hash, status: res.statusCode, body }; store.set storeKey, JSON.stringify record , "EX", DONE TTL SEC .catch err = console.error "idempotency: failed to store response", storeKey, err ; } return originalJson body ; }; // next returns void in Express, so try/catch here never sees a handler error. // Release the claim whenever the response ends without a stored record: thrown error, 5xx, res.send . res.on "close", = { if stored store.del storeKey .catch err = console.error "idempotency: failed to release claim", storeKey, err ; } ; next ; }; } Wire it up after body parsing and auth. The handler validates its input before anything reaches Stripe, and it forwards a key on the Stripe call. Your key protects your server from running the handler twice; Stripe's protects the charge if your handler runs once but your call to Stripe times out and gets retried. Stripe scopes idempotency keys to your Stripe account, not to your users, so forward the caller-scoped key — otherwise two users who both send k1 would collide at Stripe. python import express from "express"; import Redis from "ioredis"; import Stripe from "stripe"; import { z } from "zod"; const app = express ; app.use express.json ; app.use authenticate ; // sets req.userId app.use idempotency new Redis process.env.REDIS URL ; const stripe = new Stripe process.env.STRIPE SECRET KEY ; const ChargeBody = z.object { amount: z.number .int .positive , currency: z.enum "usd", "gbp" } ; app.post "/charges", async req, res = { const parsed = ChargeBody.safeParse req.body ; if parsed.success return res.status 400 .json { error: parsed.error.flatten } ; const intent = await stripe.paymentIntents.create { amount: parsed.data.amount, currency: parsed.data.currency }, { idempotencyKey: ${req.userId}:${req.idempotencyKey} }, ; res.status 201 .json { id: intent.id, status: intent.status } ; } ; The server half only works if the client reuses the key, so here is the client half: a fetch wrapper with exponential backoff, jitter so a thousand clients that timed out together don't retry together, and respect for Retry-After . export async function postWithRetry url: string, body: unknown, opts: { attempts?: number } = {} { const key = crypto.randomUUID ; // one key per logical operation, reused on every retry const attempts = opts.attempts ?? 5; for let attempt = 0; ; attempt++ { let res: Response | undefined; try { res = await fetch url, { method: "POST", headers: { "Content-Type": "application/json", "Idempotency-Key": key }, body: JSON.stringify body , signal: AbortSignal.timeout 10 000 , } ; if res.ok || res.status < 500 && res.status == 409 && res.status == 429 return res; } catch err { if attempt + 1 = attempts throw err; // network error or timeout } if attempt + 1 = attempts return res ; const retryAfterSec = Number res?.headers.get "Retry-After" ; const backoffMs = Math.min 30 000, 500 2 attempt ; const jitterMs = Math.random backoffMs; const waitMs = retryAfterSec 0 ? retryAfterSec 1000 : backoffMs + jitterMs; await new Promise r = setTimeout r, waitMs ; } } The key is generated once, outside the loop, and that line placement is the entire point of the function. If you're exposing this to an AI agent through a tool definition, generate the key inside your tool wrapper rather than asking the model to supply it. Models are not reliable about reusing the same UUID across attempts. Server crashes mid-request. The claim is written as running , then the process dies before the handler finishes. Nothing deletes the key, so every retry gets 409 until it expires. That's why the running claim gets 30 seconds and only the finished record gets 24 hours: thirty seconds is a stale lock the client can wait out, twenty-four hours is an outage for that operation. The TTL cuts the other way too. If a handler legitimately takes longer than 30 seconds, the lock expires while it's still running, a retry claims the key fresh and runs the handler again, and you're back to the duplicate you started with. Set the running TTL comfortably above your slowest handler, and if that number is uncomfortable, the handler should probably be a queued job. Multiple instances. An in-memory Map works in development and fails the moment you run two copies of the service, because the retry lands on the instance that never saw the first request. The store has to be shared: Redis, or a database table with a unique constraint on the key and INSERT ... ON CONFLICT DO NOTHING as the claim. For a lot of services the table is honestly the better choice. Scoping. The store key is idem:${userId}:${key} , and that prefix is not decoration. Without it, a client who guessed or leaked another client's key could receive the other client's stored response, order details and all. Scope by whatever identifies the caller: user ID, API key ID, tenant. The test that matters is the concurrent one. Fire the same request five times at once and assert the side effect happened exactly once. This runs against a fresh Express app on an ephemeral port, with a fake auth middleware and the in-memory KeyStore fake in place of Redis. python import assert from "node:assert/strict"; import type { AddressInfo } from "node:net"; // fresh app for the test, with a fake auth middleware: // app.use req, res, next = { req.userId = "u1"; next ; } ; let charges = 0; app.post "/charges", async req, res = { await new Promise r = setTimeout r, 20 ; // pretend to talk to Stripe charges++; res.status 201 .json { chargeId: "ch 1", amount: req.body.amount } ; } ; const srv = app.listen 0, "127.0.0.1" ; await new Promise r = srv.once "listening", r ; const base = http://127.0.0.1:${ srv.address as AddressInfo .port} ; const send = key: string, body: unknown = fetch ${base}/charges , { method: "POST", headers: { "content-type": "application/json", "Idempotency-Key": key }, body: JSON.stringify body , } ; const results = await Promise.all 1, 2, 3, 4, 5 .map = send "k1", { amount: 500 } ; assert.equal charges, 1 ; assert.deepEqual results.map r = r.status .sort , 201, 409, 409, 409, 409 ; When I ran this, the part that surprised me was the shape of that second assertion, and it took me a minute to work out why I'd expected something else. I had assumed the four losing requests would get the cached 201, because that's what "idempotent" sounds like it promises. They got 409 every time, because all five arrived during the 20 milliseconds the first was still running and there was no response to replay yet. The replay only shows up on the sixth request, sent after the first finished. That's correct, and it's what the 409 state is for, but it means a client has to treat 409 as "wait and try again" rather than as an error, or you've traded a duplicate problem for a "my request failed for no reason" problem. The other thing testing caught, which I'm less proud of. My first version wrapped await next in a try/catch and deleted the claim in the catch. The unit test passed, because it called the handler directly and the rejection came back through next . Against a real Express app, a route that throws produced a 500 from Express's error handler, the catch never ran, and the retry got 409 for the next 30 seconds. next returns nothing in Express. In Express 5 a rejected async handler goes to the error handler, not to whoever called next ; in Express 4 it's an unhandled rejection unless you add a wrapper like express-async-errors , and the request hangs until the client gives up. Either way the close listener releases the claim. It exists because of that run, and it's a good argument for testing middleware through the framework rather than around it. SET NX or a unique constraint before running the handler. Retry-After ; finished gets the stored response. If the operation is already idempotent, skip all of this. A PUT that replaces a record, a DELETE, or an upsert keyed on something the client controls gives the same state on the tenth retry as the first. You can often get idempotency from the data model too: if an order carries a client-supplied order number with a unique constraint, the second INSERT fails and you return the existing order, and that's the whole implementation. The header-and-Redis approach is for the POSTs where there's no natural key and the side effect lives outside your database, which in practice means charges, emails, and anything that calls someone else's API and can't be taken back.