{"slug": "redis-rate-limits-for-llm-api-keys-and-tenant-quotas", "title": "Redis Rate Limits for LLM API Keys and Tenant Quotas", "summary": "A developer detailed a Redis-based rate limiting architecture for LLM API keys that tracks both request counts and token consumption per tenant, arguing that token limits are what actually cap spending since requests vary in cost by orders of magnitude. The approach uses sorted sets and atomic Lua scripts for true sliding windows, derives tenant identifiers from authenticated sessions rather than request bodies, and reserves estimated token costs up front with refunds for unused capacity.", "body_md": "*This post was created with AI assistance and reviewed for accuracy before publishing.*\n\nRate limiting a normal API protects your servers. Rate limiting an LLM API protects your bank account. The difference matters, because it changes what you are counting and where the limit has to live.\n\nA conventional web endpoint costs you CPU time measured in milliseconds. A single model call can cost real money, and an agent loop can issue hundreds of them without a human ever clicking anything. One customer leaving a misconfigured retry loop running overnight is a bill, not a blip.\n\nThe instinct is to limit requests per minute. That is necessary but not sufficient, because requests vary in cost by three orders of magnitude. A one-line prompt and a 200-page document summarisation are both \"one request\".\n\nTrack both, and enforce whichever binds first:\n\n```\n// Two windows, different units, same tenant.\nawait Promise.all([\n  consume(`rl:${tenantId}:req`, 1, 60),        // 60 requests / minute\n  consume(`rl:${tenantId}:tok`, tokens, 3600), // 200k tokens / hour\n]);\n```\n\nThe request limit stops runaway loops quickly. The token limit is what actually caps spend. You need the second one before you need the first.\n\nEvery limit key must contain the tenant identifier, and that identifier must come from your authenticated session, never from the request body.\n\n``` js\n// Correct: identity comes from the verified session.\nconst key = `rl:${session.tenantId}:${operation}`;\n\n// Wrong: caller controls their own bucket, so they control their own limit.\nconst key = `rl:${req.body.tenantId}:${operation}`;\n```\n\nThat second version is not a rate limit. Anyone who reads their own network traffic can send a different tenant id and get a fresh quota, or send someone else's and exhaust theirs.\n\nInclude the operation in the key too. Embedding calls and chat completions have wildly different cost profiles, and a shared bucket means a bulk embedding job starves interactive users of the quota they are paying for.\n\nFixed windows have a well-known flaw: a caller can send a full quota at 11:59:59 and another full quota at 12:00:00, which is double the intended rate across a two-second span.\n\nA sorted set gives you a true sliding window. Each request is a member scored by timestamp; expired entries are trimmed on read.\n\n```\n-- KEYS[1] window key, ARGV: now_ms, window_ms, limit, member\nredis.call('ZREMRANGEBYSCORE', KEYS[1], 0, ARGV[1] - ARGV[2])\nlocal used = redis.call('ZCARD', KEYS[1])\nif used >= tonumber(ARGV[3]) then\n  return {0, used}\nend\nredis.call('ZADD', KEYS[1], ARGV[1], ARGV[4])\nredis.call('PEXPIRE', KEYS[1], ARGV[2])\nreturn {1, used + 1}\n```\n\nRunning this as a Lua script matters. The trim, the count, and the insert have to be one atomic unit, otherwise two concurrent requests can both read `used = limit - 1` and both proceed. Redis executes a script without interleaving other commands, which closes that race without a distributed lock.\n\nThe `PEXPIRE` on every call is deliberate. It means idle tenants expire out of memory on their own, so your key count tracks active tenants rather than every tenant you have ever had.\n\nToken counts create an ordering problem: you cannot know the true cost until the response comes back, but you have to decide before sending it.\n\nReserve an estimate up front, then correct it:\n\n``` js\nconst estimate = countPromptTokens(prompt) + maxOutputTokens;\nconst ok = await consume(`rl:${tenantId}:tok`, estimate, 3600);\nif (!ok) throw new QuotaExceeded();\n\nconst res = await model.complete(prompt);\n\n// Give back what was reserved but not used.\nconst actual = res.usage.inputTokens + res.usage.outputTokens;\nawait refund(`rl:${tenantId}:tok`, estimate - actual);\n```\n\nReserving `maxOutputTokens` is pessimistic on purpose. Most responses come in well under the ceiling, and the refund returns the difference. The alternative, charging only actual usage after the fact, lets a tenant exceed their quota by the size of whatever is in flight when they hit the limit.\n\nWhen Redis is unreachable you have to choose, and the right answer differs by limit type.\n\n| Limit | Redis down | Why | \n|---|---|---|\n| Requests per minute | Allow | Protects servers, and your servers are fine | \n| Token or spend quota | Deny | Protects money, and money does not recover | \n\nAn abuse-prevention limit failing open for thirty seconds is an acceptable risk. A spend cap failing open for thirty seconds during an agent loop is a genuinely expensive outage. Encode the difference rather than applying one blanket policy.\n\nEvery model call now waits on a Redis round trip first. That is cheap, but it is not free, and it sits directly in your user-facing path.\n\nTwo things keep it cheap. Put Redis in the same region as the service calling it, because a cross-region check can cost more than the limit saves. And pipeline the request and token checks into a single round trip rather than issuing them serially, as in the `Promise.all` above.\n\nMeasure the p99 of the check itself, separately from the model call. If it drifts upward, the usual cause is a sorted set that has grown large because the window is long and the traffic is heavy. A one-hour window on a busy tenant holds every request in memory for the full hour.\n\nStart with a per-tenant token quota on an hourly window, failing closed. That single control stops the failure mode that actually hurts, which is unbounded spend from one customer. Add per-operation buckets when a bulk job first starves your interactive traffic, and add request-per-minute limits when you see abuse rather than before.\n\nEverything here depends on the key containing an identity the caller cannot choose. Get that right and the rest is tuning.", "url": "https://wpnews.pro/news/redis-rate-limits-for-llm-api-keys-and-tenant-quotas", "canonical_source": "https://dev.to/ganeshjoshi/redis-rate-limits-for-llm-api-keys-and-tenant-quotas-3o02", "published_at": "2026-09-13 13:37:35+00:00", "updated_at": "2026-09-13 14:09:55.517512+00:00", "lang": "en", "topics": ["ai-infrastructure", "large-language-models", "ai-agents", "developer-tools", "mlops"], "entities": ["Redis"], "alternates": {"html": "https://wpnews.pro/news/redis-rate-limits-for-llm-api-keys-and-tenant-quotas", "markdown": "https://wpnews.pro/news/redis-rate-limits-for-llm-api-keys-and-tenant-quotas.md", "text": "https://wpnews.pro/news/redis-rate-limits-for-llm-api-keys-and-tenant-quotas.txt", "jsonld": "https://wpnews.pro/news/redis-rate-limits-for-llm-api-keys-and-tenant-quotas.jsonld"}}