Why Your Production API Needs Idempotency Keys (And How to Build an Engine in Node.js & Redis)

A developer outlined a production pattern for guaranteeing exact-once execution semantics in non-idempotent POST endpoints by pairing client-generated idempotency keys with a Redis-backed state machine. The approach uses atomic SET NX locking to mark a key as PROCESSING, returns 409 Conflict for concurrent retries, and caches the completed response for 24 hours, with a SHA256 hash of key plus payload to detect mismatched retries.

In a perfect network, every HTTP request arrives exactly once. In the real world, mobile clients drop connection mid-flight, timeouts hit load balancers, and frontend retry logic triggers duplicate operations. If a user clicks "Pay Now" on a $100 checkout, their connection drops, and their app automatically retries the request 3 seconds later, what happens? Without an Idempotency Engine , your API risks processing two charges for a single intent. While GET , PUT , and DELETE methods are naturally idempotent by HTTP spec, POST endpoints creating charges, generating invoices, triggering AI workflows are inherently non-idempotent. Here is how production systems guarantee exact-once execution semantics using Idempotency Keys. An idempotency key is a unique, client-generated identifier usually a v4 UUID sent in the HTTP header: Idempotency-Key: 7b9e1d84-2a3c-4e89-9102-1a4f52e39a01 When the server receives a request with an idempotency key, it enters a state machine execution loop: +-----------------------------------+ | Incoming Request + Header Key | +-----------------------------------+ | v Check Redis Cache for Key | +--------------+--------------+ | | Key Exists? Key Missing? | | +----------+----------+ v | | Set Key State: "PROCESSING" State: COMPLETE State: PROCESSING | | | v v v Execute Business Logic Return Saved Return 409 | HTTP Response Conflict v Save Response + State: "COMPLETE" key: "PROCESSING" with a short TTL e.g., 30 seconds and executes the logic. 409 Conflict or 429 Too Many Requests . Here is a clean, dependency-light middleware implementation in Node.js using Redis: python import { Request, Response, NextFunction } from 'express'; import Redis from 'ioredis'; const redis = new Redis process.env.REDIS URL ; const IDEMPOTENCY TTL = 86400; // 24 Hours in seconds export const idempotencyMiddleware = async req: Request, res: Response, next: NextFunction = { const key = req.header 'Idempotency-Key' ; // Skip if client didn't supply a key or enforce it for critical routes if key return next ; const redisKey = idempotency:${key} ; try { // Atomically set key if it doesn't exist NX with a lock timeout const acquired = await redis.set redisKey, JSON.stringify { state: 'PROCESSING' } , 'EX', 30, 'NX' ; if acquired { const cachedData = await redis.get redisKey ; if cachedData { const parsed = JSON.parse cachedData ; if parsed.state === 'PROCESSING' { return res.status 409 .json { error: 'Concurrent request in progress. Please wait.' } ; } // Return previously cached execution response return res.status parsed.statusCode .json parsed.body ; } } // Intercept res.json to capture response payload before sending to client const originalJson = res.json.bind res ; res.json = body: any = { // Save execution result to Redis for future retries redis.set redisKey, JSON.stringify { state: 'COMPLETE', statusCode: res.statusCode, body } , 'EX', IDEMPOTENCY TTL ; return originalJson body ; }; next ; } catch err { next err ; } }; What if a malicious actor sends the same Idempotency-Key with a completely different JSON payload? SHA256 Key + Payload . If the key matches but the payload hash differs, return 400 Bad Request . If your database or downstream service crashes while processing a request, do not save a 500 error as a permanent idempotent response . Always set an expiration TTL e.g., 30 seconds on the initial PROCESSING lock. If your API worker dies mid-execution, the key will naturally expire rather than permanently locking out the user from retrying. Handling retries cleanly is the difference between a brittle prototype and enterprise-ready API infrastructure. By pushing idempotency tracking to a fast Redis layer, you protect your database from duplicate writes and give your client applications a bulletproof retry strategy.