{"slug": "idempotency-keys-how-to-make-a-retried-post-do-its-job-once", "title": "Idempotency keys: how to make a retried POST do its job once", "summary": "A developer outlined a four-branch server-side implementation of the IETF draft Idempotency-Key HTTP header, which lets retried POST requests execute only once by having clients send a stable key per logical operation and servers store and replay the original response. The approach requires an atomic key claim to prevent duplicate execution across instances, returns 409 Conflict with Retry-After while an original request is still in flight, and rejects same-key/different-body retries with 422 after comparing a canonicalized body hash. The developer noted that AI agents retry more aggressively than human-written clients, making idempotency handling more pressing for APIs they call.", "body_md": "The client did nothing wrong, which is the part that took me a while to accept. It sent a request, got no answer, and sent it again, which is exactly what every HTTP client is told to do. The problem is that \"no answer\" covers two situations that look identical from the client's side: either the request never arrived, or it arrived, did its work, and the response got lost on the way back. The client can't tell which. So it retries, which is correct, and the server does the work twice, which is not.\n\nTimeouts and flaky networks have always caused this. What's newer is AI agents calling your API, and in my experience they retry harder than any human-written client, because a model that sees an error will usually just try again with the same arguments until something works.\n\nAn operation is idempotent if doing it once and doing it five times leave the system in the same state and give the client the same answer. \"Set the user's email to X\" is idempotent. \"Add 5 to the balance\" is not.\n\nHTTP sorts its methods this way already. GET, HEAD, PUT and DELETE are defined as idempotent. Deleting something already gone returns 404 rather than 204, so the response changes between calls, but the state doesn't, and the state is what the definition is about.\n\nPOST has no such guarantee, and can't really, because POST means \"process this\" and the server decides what processing means. Create an order, send an email, charge a card. The awkward thing is that the operations that actually matter in an API, the ones involving money or messages, are nearly all POSTs, so the protocol gives you nothing where you need it most.\n\nThe fix has been in payment APIs for years and is now an IETF draft, \"The Idempotency-Key HTTP Header Field,\" from the HTTPAPI working group. It's still a draft and hasn't become an RFC, but the design is simple enough that it's the convention regardless.\n\nThe client generates a unique key, usually a UUID, for each logical operation. \"Charge this customer $5 for this order\" gets one key, and every retry of that operation sends the same key in an `Idempotency-Key` header. The server uses it to recognise a request it has already seen and return the original answer instead of doing the work again.\n\nThe word that matters there is \"logical.\" The key identifies what the user is trying to accomplish, not the individual HTTP request. I've seen clients generate a fresh key inside the retry loop, so every attempt carries a different one, and at that point you've implemented the header without any of the behaviour.\n\nEach unsafe request with a key lands in one of four states, and the implementation is basically these four branches. GET and HEAD skip the whole thing; they're already idempotent and storing their responses would just be a cache with worse semantics.\n\nFirst arrival: claim the key, run the handler, store the response status and body. The claim has to be atomic, because two copies can arrive within the same millisecond on two instances, and if the claim is a read followed by a write, both read \"nothing here\" and both run.\n\nRetry after the first finished: return the stored response, run nothing. I add an `Idempotent-Replayed: true` header so I can tell real responses from replays in logs.\n\nRetry while the first is still running: there's nothing to replay yet. Return 409 Conflict with `Retry-After` and let the client ask again in a second. This feels like it should be rare and it isn't, for reasons I'll get to in testing.\n\nSame key, different body: something is wrong on the client. Hash the body on the first request, store the hash, compare on every retry. A mismatch gets 422 Unprocessable Content, which is what the draft recommends. Hash a canonical form of the body, with object keys sorted, or a client that re-serialises `{\"a\":1,\"b\":2}` as `{\"b\":2,\"a\":1}` on retry gets a 422 it didn't earn.\n\nKeys expire. 24 hours is the usual window, long enough for any realistic retry schedule and short enough that the store doesn't grow forever.\n\nThe middleware depends on a small interface over Redis rather than the ioredis client directly, so tests can swap in a fake. It also makes it obvious the whole thing needs three commands.\n\n``` python\nimport { createHash } from \"node:crypto\";\nimport type { Request, Response, NextFunction } from \"express\";\n\ndeclare global {\n  namespace Express {\n    interface Request {\n      userId?: string;         // set by auth middleware\n      idempotencyKey?: string; // set here, for handlers to forward downstream\n    }\n  }\n}\n\n// The subset of ioredis we use. Lets tests pass a fake.\nexport interface KeyStore {\n  set(key: string, value: string, ...args: (string | number)[]): Promise<\"OK\" | null>;\n  get(key: string): Promise<string | null>;\n  del(key: string): Promise<number>;\n}\n\ntype IdempotencyRecord =\n  | { state: \"running\"; hash: string }\n  | { state: \"done\"; hash: string; status: number; body: unknown };\n\nconst SAFE_METHODS = new Set([\"GET\", \"HEAD\", \"OPTIONS\"]);\nconst RUNNING_TTL_SEC = 30;          // lock held while the handler runs; exceed it and a retry runs again\nconst DONE_TTL_SEC = 60 * 60 * 24;   // replay window, 24h\n\n// Sort object keys so {\"a\":1,\"b\":2} and {\"b\":2,\"a\":1} hash the same.\nexport function canonicalJson(value: unknown): string {\n  if (Array.isArray(value)) return `[${value.map(canonicalJson).join(\",\")}]`;\n  if (value && typeof value === \"object\") {\n    const entries = Object.keys(value).sort()\n      .map((k) => `${JSON.stringify(k)}:${canonicalJson((value as Record<string, unknown>)[k])}`);\n    return `{${entries.join(\",\")}}`;\n  }\n  return JSON.stringify(value ?? null);\n}\n\nexport function hashBody(body: unknown): string {\n  return createHash(\"sha256\").update(canonicalJson(body)).digest(\"hex\");\n}\n```\n\nThe claim is one Redis command. `SET ... NX` only writes if the key doesn't exist and returns `null` otherwise, so the first request wins and every other copy sees `null`. That one flag does all the concurrency work here, which I find slightly unnerving but is also why it's reliable.\n\n```\nexport function idempotency(store: KeyStore) {\n  return async function middleware(req: Request, res: Response, next: NextFunction) {\n    const key = req.header(\"Idempotency-Key\");\n    if (!key || SAFE_METHODS.has(req.method)) return next();\n    if (!req.userId) return res.status(401).end();\n\n    req.idempotencyKey = key;\n    const storeKey = `idem:${req.userId}:${key}`;\n    const hash = hashBody(req.body);\n\n    // Atomic claim. Only one concurrent request wins.\n    const claimed = await store.set(\n      storeKey, JSON.stringify({ state: \"running\", hash }), \"NX\", \"EX\", RUNNING_TTL_SEC,\n    );\n```\n\nIf the claim failed, read what's there and pick one of the three replies.\n\n``` js\n    if (claimed === null) {\n      const existing = JSON.parse((await store.get(storeKey)) ?? \"null\") as IdempotencyRecord | null;\n      if (!existing) return res.status(409).set(\"Retry-After\", \"1\").end(); // lock expired between SET and GET\n      if (existing.hash !== hash) {\n        return res.status(422).json({ error: \"Idempotency-Key reused with a different request body\" });\n      }\n      if (existing.state === \"running\") {\n        return res.status(409).set(\"Retry-After\", \"1\").json({ error: \"Request with this key is still in progress\" });\n      }\n      return res.status(existing.status).set(\"Idempotent-Replayed\", \"true\").json(existing.body);\n    }\n```\n\nIf the claim succeeded, wrap `res.json` so the response gets stored when the handler sends it. Anything that ends the response without storing a record, which covers thrown errors, 5xx, and handlers that use `res.send`, releases the claim so the client can retry. A 4xx is stored and replayed on purpose: the same body will fail validation the same way, so there's no reason to run it again to find out.\n\n```\n    // Store the response when the handler sends it. 5xx is not stored, so the client may retry it.\n    let stored = false;\n    const originalJson = res.json.bind(res);\n    res.json = (body: unknown) => {\n      if (res.statusCode < 500) {\n        stored = true;\n        const record: IdempotencyRecord = { state: \"done\", hash, status: res.statusCode, body };\n        store.set(storeKey, JSON.stringify(record), \"EX\", DONE_TTL_SEC)\n          .catch((err) => console.error(\"idempotency: failed to store response\", storeKey, err));\n      }\n      return originalJson(body);\n    };\n\n    // next() returns void in Express, so try/catch here never sees a handler error.\n    // Release the claim whenever the response ends without a stored record: thrown error, 5xx, res.send().\n    res.on(\"close\", () => {\n      if (!stored) store.del(storeKey).catch((err) => console.error(\"idempotency: failed to release claim\", storeKey, err));\n    });\n\n    next();\n  };\n}\n```\n\nWire it up after body parsing and auth. The handler validates its input before anything reaches Stripe, and it forwards a key on the Stripe call. Your key protects your server from running the handler twice; Stripe's protects the charge if your handler runs once but your call to Stripe times out and gets retried. Stripe scopes idempotency keys to your Stripe account, not to your users, so forward the caller-scoped key — otherwise two users who both send `k1` would collide at Stripe.\n\n``` python\nimport express from \"express\";\nimport Redis from \"ioredis\";\nimport Stripe from \"stripe\";\nimport { z } from \"zod\";\n\nconst app = express();\napp.use(express.json());\napp.use(authenticate);                  // sets req.userId\napp.use(idempotency(new Redis(process.env.REDIS_URL)));\n\nconst stripe = new Stripe(process.env.STRIPE_SECRET_KEY!);\nconst ChargeBody = z.object({ amount: z.number().int().positive(), currency: z.enum([\"usd\", \"gbp\"]) });\n\napp.post(\"/charges\", async (req, res) => {\n  const parsed = ChargeBody.safeParse(req.body);\n  if (!parsed.success) return res.status(400).json({ error: parsed.error.flatten() });\n  const intent = await stripe.paymentIntents.create(\n    { amount: parsed.data.amount, currency: parsed.data.currency },\n    { idempotencyKey: `${req.userId}:${req.idempotencyKey}` },\n  );\n  res.status(201).json({ id: intent.id, status: intent.status });\n});\n```\n\nThe server half only works if the client reuses the key, so here is the client half: a fetch wrapper with exponential backoff, jitter so a thousand clients that timed out together don't retry together, and respect for `Retry-After`.\n\n```\nexport async function postWithRetry(url: string, body: unknown, opts: { attempts?: number } = {}) {\n  const key = crypto.randomUUID(); // one key per logical operation, reused on every retry\n  const attempts = opts.attempts ?? 5;\n\n  for (let attempt = 0; ; attempt++) {\n    let res: Response | undefined;\n    try {\n      res = await fetch(url, {\n        method: \"POST\",\n        headers: { \"Content-Type\": \"application/json\", \"Idempotency-Key\": key },\n        body: JSON.stringify(body),\n        signal: AbortSignal.timeout(10_000),\n      });\n      if (res.ok || (res.status < 500 && res.status !== 409 && res.status !== 429)) return res;\n    } catch (err) {\n      if (attempt + 1 >= attempts) throw err; // network error or timeout\n    }\n    if (attempt + 1 >= attempts) return res!;\n\n    const retryAfterSec = Number(res?.headers.get(\"Retry-After\"));\n    const backoffMs = Math.min(30_000, 500 * 2 ** attempt);\n    const jitterMs = Math.random() * backoffMs;\n    const waitMs = retryAfterSec > 0 ? retryAfterSec * 1000 : backoffMs + jitterMs;\n    await new Promise((r) => setTimeout(r, waitMs));\n  }\n}\n```\n\nThe key is generated once, outside the loop, and that line placement is the entire point of the function. If you're exposing this to an AI agent through a tool definition, generate the key inside your tool wrapper rather than asking the model to supply it. Models are not reliable about reusing the same UUID across attempts.\n\nServer crashes mid-request. The claim is written as `running`, then the process dies before the handler finishes. Nothing deletes the key, so every retry gets 409 until it expires. That's why the running claim gets 30 seconds and only the finished record gets 24 hours: thirty seconds is a stale lock the client can wait out, twenty-four hours is an outage for that operation. The TTL cuts the other way too. If a handler legitimately takes longer than 30 seconds, the lock expires while it's still running, a retry claims the key fresh and runs the handler again, and you're back to the duplicate you started with. Set the running TTL comfortably above your slowest handler, and if that number is uncomfortable, the handler should probably be a queued job.\n\nMultiple instances. An in-memory `Map` works in development and fails the moment you run two copies of the service, because the retry lands on the instance that never saw the first request. The store has to be shared: Redis, or a database table with a unique constraint on the key and `INSERT ... ON CONFLICT DO NOTHING` as the claim. For a lot of services the table is honestly the better choice.\n\nScoping. The store key is `idem:${userId}:${key}`, and that prefix is not decoration. Without it, a client who guessed or leaked another client's key could receive the other client's stored response, order details and all. Scope by whatever identifies the caller: user ID, API key ID, tenant.\n\nThe test that matters is the concurrent one. Fire the same request five times at once and assert the side effect happened exactly once. This runs against a fresh Express app on an ephemeral port, with a fake auth middleware and the in-memory `KeyStore` fake in place of Redis.\n\n``` python\nimport assert from \"node:assert/strict\";\nimport type { AddressInfo } from \"node:net\";\n\n// fresh app for the test, with a fake auth middleware:\n// app.use((req, _res, next) => { req.userId = \"u1\"; next(); });\n\nlet charges = 0;\napp.post(\"/charges\", async (req, res) => {\n  await new Promise((r) => setTimeout(r, 20)); // pretend to talk to Stripe\n  charges++;\n  res.status(201).json({ chargeId: \"ch_1\", amount: req.body.amount });\n});\n\nconst srv = app.listen(0, \"127.0.0.1\");\nawait new Promise((r) => srv.once(\"listening\", r));\nconst base = `http://127.0.0.1:${(srv.address() as AddressInfo).port}`;\nconst send = (key: string, body: unknown) =>\n  fetch(`${base}/charges`, {\n    method: \"POST\",\n    headers: { \"content-type\": \"application/json\", \"Idempotency-Key\": key },\n    body: JSON.stringify(body),\n  });\n\nconst results = await Promise.all([1, 2, 3, 4, 5].map(() => send(\"k1\", { amount: 500 })));\nassert.equal(charges, 1);\nassert.deepEqual(results.map((r) => r.status).sort(), [201, 409, 409, 409, 409]);\n```\n\nWhen I ran this, the part that surprised me was the shape of that second assertion, and it took me a minute to work out why I'd expected something else. I had assumed the four losing requests would get the cached 201, because that's what \"idempotent\" sounds like it promises. They got 409 every time, because all five arrived during the 20 milliseconds the first was still running and there was no response to replay yet. The replay only shows up on the sixth request, sent after the first finished. That's correct, and it's what the 409 state is for, but it means a client has to treat 409 as \"wait and try again\" rather than as an error, or you've traded a duplicate problem for a \"my request failed for no reason\" problem.\n\nThe other thing testing caught, which I'm less proud of. My first version wrapped `await next()` in a try/catch and deleted the claim in the catch. The unit test passed, because it called the handler directly and the rejection came back through `next`. Against a real Express app, a route that throws produced a 500 from Express's error handler, the catch never ran, and the retry got 409 for the next 30 seconds. `next()` returns nothing in Express. In Express 5 a rejected async handler goes to the error handler, not to whoever called `next`; in Express 4 it's an unhandled rejection unless you add a wrapper like `express-async-errors`, and the request hangs until the client gives up. Either way the `close` listener releases the claim. It exists because of that run, and it's a good argument for testing middleware through the framework rather than around it.\n\n`SET NX` or a unique constraint) before running the handler.`Retry-After`; finished gets the stored response.\nIf the operation is already idempotent, skip all of this. A PUT that replaces a record, a DELETE, or an upsert keyed on something the client controls gives the same state on the tenth retry as the first. You can often get idempotency from the data model too: if an order carries a client-supplied order number with a unique constraint, the second INSERT fails and you return the existing order, and that's the whole implementation. The header-and-Redis approach is for the POSTs where there's no natural key and the side effect lives outside your database, which in practice means charges, emails, and anything that calls someone else's API and can't be taken back.", "url": "https://wpnews.pro/news/idempotency-keys-how-to-make-a-retried-post-do-its-job-once", "canonical_source": "https://dev.to/rishi_makwana/idempotency-keys-how-to-make-a-retried-post-do-its-job-once-3620", "published_at": "2026-10-25 04:32:14+00:00", "updated_at": "2026-10-25 04:42:25.310928+00:00", "lang": "en", "topics": ["ai-agents", "developer-tools", "ai-tools"], "entities": ["IETF", "HTTPAPI Working Group"], "also_reported_by": [], "alternates": {"html": "https://wpnews.pro/news/idempotency-keys-how-to-make-a-retried-post-do-its-job-once", "markdown": "https://wpnews.pro/news/idempotency-keys-how-to-make-a-retried-post-do-its-job-once.md", "text": "https://wpnews.pro/news/idempotency-keys-how-to-make-a-retried-post-do-its-job-once.txt", "jsonld": "https://wpnews.pro/news/idempotency-keys-how-to-make-a-retried-post-do-its-job-once.jsonld"}}