Why Your Production API Needs Idempotency Keys (And How to Build an Engine in Node.js & Redis)

In a perfect network, every HTTP request arrives exactly once. In the real world, mobile clients drop connection mid-flight, timeouts hit load balancers, and frontend retry logic triggers duplicate operations.

If a user clicks "Pay Now" on a $100 checkout, their connection drops, and their app automatically retries the request 3 seconds later, what happens?

Without an Idempotency Engine, your API risks processing two charges for a single intent.

While GET, PUT, and DELETE methods are naturally idempotent by HTTP spec, POST endpoints (creating charges, generating invoices, triggering AI workflows) are inherently non-idempotent.

Here is how production systems guarantee exact-once execution semantics using Idempotency Keys.

An idempotency key is a unique, client-generated identifier (usually a v4 UUID) sent in the HTTP header:

Idempotency-Key: 7b9e1d84-2a3c-4e89-9102-1a4f52e39a01

When the server receives a request with an idempotency key, it enters a state machine execution loop:

                  +-----------------------------------+
                  |   Incoming Request + Header Key   |
                  +-----------------------------------+
                                    |
                                    v
                       [ Check Redis Cache for Key ]
                                    |
                     +--------------+--------------+
                     |                             |
             (Key Exists?)                   (Key Missing?)
                     |                             |
          +----------+----------+                  v
          |                     |       [ Set Key State: "PROCESSING" ]
   (State: COMPLETE)   (State: PROCESSING)         |
          |                     |                  v
          v                     v       [ Execute Business Logic ]
   [ Return Saved ]      [ Return 409 ]            |
   [ HTTP Response ]     [ Conflict ]              v
                                        [ Save Response + State: "COMPLETE" ]

key: "PROCESSING" with a short TTL (e.g., 30 seconds) and executes the logic.409 Conflict or 429 Too Many Requests. Here is a clean, dependency-light middleware implementation in Node.js using Redis:

import { Request, Response, NextFunction } from 'express';
import Redis from 'ioredis';

const redis = new Redis(process.env.REDIS_URL);
const IDEMPOTENCY_TTL = 86400; // 24 Hours in seconds

export const idempotencyMiddleware = async (
  req: Request, 
  res: Response, 
  next: NextFunction
) => {
  const key = req.header('Idempotency-Key');

  // Skip if client didn't supply a key (or enforce it for critical routes)
  if (!key) return next();

  const redisKey = `idempotency:${key}`;

  try {
    // Atomically set key if it doesn't exist (NX) with a lock timeout
    const acquired = await redis.set(redisKey, JSON.stringify({ state: 'PROCESSING' }), 'EX', 30, 'NX');

    if (!acquired) {
      const cachedData = await redis.get(redisKey);

      if (cachedData) {
        const parsed = JSON.parse(cachedData);
        if (parsed.state === 'PROCESSING') {
          return res.status(409).json({ error: 'Concurrent request in progress. Please wait.' });
        }
        // Return previously cached execution response
        return res.status(parsed.statusCode).json(parsed.body);
      }
    }

    // Intercept res.json to capture response payload before sending to client
    const originalJson = res.json.bind(res);
    res.json = (body: any) => {
      // Save execution result to Redis for future retries
      redis.set(
        redisKey, 
        JSON.stringify({ state: 'COMPLETE', statusCode: res.statusCode, body }), 
        'EX', 
        IDEMPOTENCY_TTL
      );
      return originalJson(body);
    };

    next();
  } catch (err) {
    next(err);
  }
};

What if a malicious actor sends the same Idempotency-Key with a completely different JSON payload?

SHA256(Key + Payload)). If the key matches but the payload hash differs, return 400 Bad Request. If your database or downstream service crashes while processing a request, do not save a 500 error as a permanent idempotent response.

Always set an expiration TTL (e.g., 30 seconds) on the initial PROCESSING lock. If your API worker dies mid-execution, the key will naturally expire rather than permanently locking out the user from retrying.

Handling retries cleanly is the difference between a brittle prototype and enterprise-ready API infrastructure. By pushing idempotency tracking to a fast Redis layer, you protect your database from duplicate writes and give your client applications a bulletproof retry strategy.