Too Many Req: A Bucket List Guide to Building a Rate Limiter

Developer Maneshwar is building git-lrc, a free and source-available Micro AI code reviewer that runs on every commit. In a technical blog post, he provides a guide to designing a rate limiter, covering fixed window counting, the pitfalls of database and in-memory counters, and the token bucket algorithm used by AWS and Stripe.

Hello, I'm Maneshwar. I'm building git-lrc, a Micro AI code reviewer that runs on every commit. It is free and source-available on Github. Star git-lrc to help devs discover the project. Do give it a try and share your feedback. Every serious API will eventually tell you to sit down and be quiet. Hammer GitHub, Stripe, or AWS a little too eagerly and your requests start bouncing back with a polite but firm 429 . I always found that fascinating, so let's build the thing that says no. By the end of this post we'll have designed a rate limiter that actually holds up when you put it in front of real traffic, and I promise to only make a reasonable number of bucket puns along the way. A rate limiter does one job: it decides how many requests a client is allowed to make in a given window of time. It protects your system from getting flattened, and it keeps one greedy user from eating everyone else's lunch. Simple idea. Surprisingly spicy implementation. Let's build it up piece by piece, the way you'd actually reason through it in an interview or a design doc. Before writing a single line, let's agree on what "good" looks like. Here's my wishlist: 429 Too Many Requests Cool. Now let's start naive and let reality punch us in the face a few times. The simplest thing that could possibly work is fixed window counting . Chop time into neat one-minute slices. Give each user a counter. Every request bumps the counter by one. Hit the limit, get rejected, and the counter resets when the next window starts. user makes a request count = get user id how many so far this minute? if count = 100: reject 429, come back later else: increment user id allow Clean. Readable. You could explain it to a rubber duck. So where do we keep this counter? Your first instinct might be the database. Please don't. We'd be adding a write to the database on every request , which means the thing we built to protect our system is now quietly overloading it. That is peak "I have brought peace, freedom, and a full table scan." Okay, database is out. What about keeping counters in memory on the server ? Blazing fast. Love it. Except it only works if you have exactly one server, and nobody runs one server. The moment you scale out, each box keeps its own private counter. A sneaky user sends 100 requests to Server A and 100 to Server B and walks away with 200 requests per minute while your limit says 100. Whoops. What we actually want is somewhere that is memory-fast and shared across every server. That's Redis https://redis.io/ . It's an in-memory data store, it hands us atomic counter primitives like INCR , and it can expire keys automatically so windows reset on their own. This is why Redis shows up in basically every rate limiter design ever drawn on a whiteboard. So far so good. Now let me ruin it. Fixed windows have a nasty edge case hiding right at the seams. Picture a limit of 100 requests per minute. A user fires 100 requests in the last 10 seconds of one minute, then another 100 in the first 10 seconds of the next minute. Each window is technically within the limit. Both are 100 or under. But zoom out and you'll see 200 requests in a 20 second span, which is very much not the spirit of "100 per minute." This happens at every window boundary, and once someone notices the pattern, they will absolutely abuse it. The counter has no memory across the boundary, so it cannot see the burst spanning two windows. We need an algorithm that thinks in terms of a smooth rate rather than hard resets. Enter the token bucket , the algorithm quietly powering the limits at places like AWS https://docs.aws.amazon.com/ and Stripe https://docs.stripe.com/rate-limits . Here's the mental model, and yes, it is literally a bucket. Imagine a bucket that holds tokens. Tokens drip in at a steady rate. Every request has to grab one token to pass. No tokens left? Request gets rejected. That's it. refill based on time passed since we last looked elapsed = now - last refill tokens = min capacity, tokens + elapsed refill rate last refill = now if tokens = 1: tokens -= 1 allow else: reject 429, and tell them when to retry Here's the diagram version of that decision: Watch how this fixes our boundary nightmare. Set the bucket capacity to 100 and the refill rate to 100 per minute. During quiet stretches, tokens pile up toward the cap. When a burst comes in, the user spends whatever tokens they've saved, but they can never outrun the refill rate over the long haul. No matter how they time things around a boundary, they cannot conjure tokens that were never added. That's the whole trick, and it's genuinely elegant. Two knobs control everything: Capacity 100 with a refill of 100 per minute means a user can fire up to 100 requests instantly if they've been idle, but long term they're pinned to 100 per minute. Bursty when it's calm, strict when it counts. Chef's kiss. Are there other algorithms? Sure. Sliding window logs, sliding window counters, leaky buckets, they each solve the boundary problem their own way. But token bucket hits the sweet spot between "simple enough to actually implement correctly" and "good enough for almost everyone." For a deeper rabbit hole, the token bucket writeup on Wikipedia https://en.wikipedia.org/wiki/Token bucket is a decent start. We've got the algorithm. Now, architecturally, where do we run it? Three options, each with a personality. Which one wins? It genuinely depends. If you already run an API gateway doing auth, tucking rate limiting in there is a no-brainer. If you need some exotic custom algorithm, server side gives you room to move. But for most systems, middleware is the sweet spot: control and operational sanity without soldering the limiter to your business code. Let's go with middleware and sketch the flow. We'll store the rules "premium gets 1000 per hour, free gets 100 per hour" in a configuration service. The middleware reads those rules, keeps token bucket state in Redis, and makes the call on every incoming request. When a request lands, the middleware figures out who the user is, pulls their bucket from Redis, and checks for a spare token. Token available? Decrement it and pass the request along. Bucket empty? Return a 429 with a Retry-After https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/Retry-After header so the client knows exactly when to come knocking again. Be a good host. Tell your guests when the kitchen reopens xD This is lovely for one rate limiting server. Then, as always, scaling shows up to spoil the party. Run multiple rate limiter instances against the same Redis and you can hit a classic race condition . Here's the exact sequence that loses a count: Both servers read 3. Both decide the request is fine. Both write 4. We just quietly lost a count, and the counter now lies to us. Do this enough times under load and your "100 per minute" limit turns into "somewhere around 100, we think, on a good day." Not exactly the airtight guarantee we promised. The problem is that read, check, and write are three separate steps, and another server can sneak in between them. The fix is to make read plus check plus write a single indivisible operation, so nobody can wedge themselves in the middle. Redis lets us do this with Lua scripts https://redis.io/docs/latest/develop/interact/programmability/eval-intro/ , which run atomically on the server. The whole check-and-decrement happens as one unit, and the race condition simply cannot occur. -- KEYS 1 = bucket key -- ARGV 1 = capacity, ARGV 2 = refill rate, ARGV 3 = now local tokens = tonumber redis.call "GET", KEYS 1 or ARGV 1 if tokens = 1 then redis.call "DECR", KEYS 1 return 1 -- allowed else return 0 -- rejected, send a 429 end That snippet is simplified to keep the point front and center. A production version would compute the refill from elapsed time and store the last-refill timestamp too. Atomic operations turn our shaky-under-load counter into something you can actually trust across a fleet of servers. This is the difference between a rate limiter that works in the demo and one that works on Black Friday. We covered the load-bearing bits, but a real production rate limiter opens up a bunch of fun follow-up questions worth chewing on: Each of those is a great whiteboard prompt on its own, and honestly, "fail open vs fail closed" alone has sparked some very heated lunch debates. So there's our bucket list, completed. We started with a naive counter, watched it leak requests at window boundaries, upgraded to a token bucket, argued about where to run it, moved the state into Redis so servers could agree, then made the whole thing atomic so scaling couldn't corrupt our counts. That's the core of nearly every rate limiter you'll ever meet in the wild. Next time an API hits you with a 429 , you'll know there's a little bucket of tokens somewhere, freshly emptied, quietly telling you to hold your horses. If you build one, or you have strong opinions on fail open vs fail closed, drop a comment. I'll try to reply before you hit my rate limit. AI agents write code fast. They also silently remove logic, change behavior, and introduce bugs — without telling you. You often find out in production. git-lrc fixes this. It hooks into git commit and reviews every diff before it lands. 60-second setup. Completely free. Any feedback or contributors are welcome It's online, source-available, and ready for anyone to use. ⭐ Star it on GitHub: | 🇩🇰 Dansk https://github.com/HexmosTech/git-lrc/readme/README.da.md | 🇪🇸 Español https://github.com/HexmosTech/git-lrc/readme/README.es.md | 🇮🇷 Farsi https://github.com/HexmosTech/git-lrc/readme/README.fa.md | 🇫🇮 Suomi https://github.com/HexmosTech/git-lrc/readme/README.fi.md | 🇯🇵 日本語 https://github.com/HexmosTech/git-lrc/readme/README.ja.md | 🇳🇴 Norsk https://github.com/HexmosTech/git-lrc/readme/README.nn.md | 🇵🇹 Português https://github.com/HexmosTech/git-lrc/readme/README.pt.md | 🇷🇺 Русский https://github.com/HexmosTech/git-lrc/readme/README.ru.md | 🇦🇱 Shqip https://github.com/HexmosTech/git-lrc/readme/README.sq.md | 🇨🇳 中文 https://github.com/HexmosTech/git-lrc/readme/README.zh.md | 🇮🇳 हिन्दी https://github.com/HexmosTech/git-lrc/readme/README.hi.md | GenAI today is a race car without brakes . It accelerates fast -- you describe something, and large blocks of code appear instantly. But AI agents silently break things : they remove logic, relax constraints, introduce expensive cloud calls, leak credentials, and change behavior -- without telling you. You often find out in production. git-lrc is your braking system. It hooks into git commit and runs an AI review on every diff In short, git-lrc helps Prevent Outages, Breaches, and Technical Debt Before They Happen At a glance: 10 risk categories https://github.com/HexmosTech/git-lrc what-git-lrc-checks-for · 100+ failure patterns tracked https://github.com/HexmosTech/git-lrc what-git-lrc-checks-for · every commit…