Practical Rate Limiting for Free Public AI Endpoints Without Annoying Real Users

If you ship a free, no-signup AI web tool, a naive rate limit like "10 requests per minute per IP" almost always fails in both directions.

Real users on shared mobile carrier NATs or office Wi-Fi hit false-positive blocks after two clicks, while simple rotating-IP scripts burn through your token budget in minutes.

Here is a lightweight, practical rate-limiting pattern that protects LLM endpoints without ruining the first-time experience for genuine visitors.

Counting only requests treats a 40-character prompt the same as a 4,000-character prompt. Instead, enforce two lightweight checks in Redis (or an in-memory store for single-instance apps):

type LimitCheck = {
  allowed: boolean;
  retryAfterSec: number;
};

type RedisCounter = {
  incr: (k: string) => Promise<number>;
  incrby: (k: string, n: number) => Promise<number>;
  expire: (k: string, s: number) => Promise<void>;
  ttl: (k: string) => Promise<number>;
};

async function checkDualLimit(
  key: string,
  charCost: number,
  redis: RedisCounter
): Promise<LimitCheck> {
  const burstKey = `rl:burst:${key}`;
  const budgetKey = `rl:chars:${key}`;

  const burstCount = await redis.incr(burstKey);
  if (burstCount === 1) await redis.expire(burstKey, 30);

  if (burstCount > 5) {
    const ttl = Math.max(1, await redis.ttl(burstKey));
    return { allowed: false, retryAfterSec: ttl };
  }

  const totalChars = await redis.incrby(budgetKey, charCost);
  if (totalChars === charCost) await redis.expire(budgetKey, 3600);

  if (totalChars > 25000) {
    const ttl = Math.max(1, await redis.ttl(budgetKey));
    return { allowed: false, retryAfterSec: ttl };
  }

  return { allowed: true, retryAfterSec: 0 };
}

On mobile networks and dual-stack connections, a single user's IPv6 address can rotate across requests, while thousands of legitimate visitors can share one IPv4 exit node.

Two small adjustments reduce both abuse and false positives:

/64 prefix/128 host address.HttpOnly anonymous session token Every rejected invalid request before your LLM call saves money and keeps your queue fast for real users:

maxInputChars limit right in the route handler.(normalizedPrompt + mode + locale) for 60 seconds. If a user accidentally hits Submit twice—or refreshes immediately—serve the cached response without counting it as a second model call.Origin / Sec-Fetch-Site headers429 Payloads With Retry-After A plain "Too Many Requests" string leaves users guessing whether the app is broken or permanently blocked. Return a standard Retry-After header plus structured JSON:

return Response.json(
  {
    error: "rate_limited",
    message: "You've sent several requests quickly. Please wait a moment and try again.",
    retryAfterSec: result.retryAfterSec,
  },
  {
    status: 429,
    headers: {
      "Retry-After": String(result.retryAfterSec),
    },
  }
);

On the frontend, disable the submit button with a live countdown (Try again in 14s) instead of letting frustrated users hammer the button and extend their own cooldown.

Good rate limiting on public AI endpoints isn't about locking everything down with heavy CAPTCHAs on the first visit. Combining burst + character budgets, /64 IPv6 normalization, short prompt-hash deduplication, and clear 429 countdowns stops most accidental and automated waste while keeping the experience smooth for real visitors.

How are you protecting free LLM endpoints in production—token buckets, anonymous session cookies, or Edge middleware?