Hello, I'm Maneshwar. I'm building git-lrc, a Micro AI code reviewer that runs on every commit. It is free and source-available on Github. Star git-lrc to help devs discover the project. Do give it a try and share your feedback.
Every serious API will eventually tell you to sit down and be quiet.
Hammer GitHub, Stripe, or AWS a little too eagerly and your requests start bouncing back with a polite but firm 429
.
I always found that fascinating, so let's build the thing that says no.
By the end of this post we'll have designed a rate limiter that actually holds up when you put it in front of real traffic, and I promise to only make a reasonable number of bucket puns along the way.
A rate limiter does one job: it decides how many requests a client is allowed to make in a given window of time.
It protects your system from getting flattened, and it keeps one greedy user from eating everyone else's lunch.
Simple idea. Surprisingly spicy implementation.
Let's build it up piece by piece, the way you'd actually reason through it in an interview or a design doc.
Before writing a single line, let's agree on what "good" looks like.
Here's my wishlist:
429 Too Many Requests
Cool. Now let's start naive and let reality punch us in the face a few times.
The simplest thing that could possibly work is fixed window counting. Chop time into neat one-minute slices.
Give each user a counter.
Every request bumps the counter by one.
Hit the limit, get rejected, and the counter resets when the next window starts.
count = get(user_id) # how many so far this minute?
if count >= 100:
reject() # 429, come back later
else:
increment(user_id)
allow()
Clean. Readable. You could explain it to a rubber duck. So where do we keep this counter?
Your first instinct might be the database.
Please don't.
We'd be adding a write to the database on every request, which means the thing we built to protect our system is now quietly over it.
That is peak "I have brought peace, freedom, and a full table scan."
Okay, database is out.
What about keeping counters in memory on the server? Blazing fast. Love it.
Except it only works if you have exactly one server, and nobody runs one server.
The moment you scale out, each box keeps its own private counter.
A sneaky user sends 100 requests to Server A and 100 to Server B and walks away with 200 requests per minute while your limit says 100. Whoops.
What we actually want is somewhere that is memory-fast and shared across every server.
That's Redis. It's an in-memory data store, it hands us atomic counter primitives like INCR
, and it can expire keys automatically so windows reset on their own.
This is why Redis shows up in basically every rate limiter design ever drawn on a whiteboard.
So far so good. Now let me ruin it.
Fixed windows have a nasty edge case hiding right at the seams.
Picture a limit of 100 requests per minute.
A user fires 100 requests in the last 10 seconds of one minute, then another 100 in the first 10 seconds of the next minute.
Each window is technically within the limit. Both are 100 or under.
But zoom out and you'll see 200 requests in a 20 second span, which is very much not the spirit of "100 per minute."
This happens at every window boundary, and once someone notices the pattern, they will absolutely abuse it.
The counter has no memory across the boundary, so it cannot see the burst spanning two windows.
We need an algorithm that thinks in terms of a smooth rate rather than hard resets.
Enter the token bucket, the algorithm quietly powering the limits at places like AWS and Stripe.
Here's the mental model, and yes, it is literally a bucket.
Imagine a bucket that holds tokens. Tokens drip in at a steady rate. Every request has to grab one token to pass. No tokens left? Request gets rejected. That's it.
elapsed = now - last_refill
tokens = min(capacity, tokens + elapsed * refill_rate)
last_refill = now
if tokens >= 1:
tokens -= 1
allow()
else:
reject() # 429, and tell them when to retry
Here's the diagram version of that decision:
Watch how this fixes our boundary nightmare.
Set the bucket capacity to 100 and the refill rate to 100 per minute.
During quiet stretches, tokens pile up toward the cap.
When a burst comes in, the user spends whatever tokens they've saved, but they can never outrun the refill rate over the long haul.
No matter how they time things around a boundary, they cannot conjure tokens that were never added.
That's the whole trick, and it's genuinely elegant. Two knobs control everything:
Capacity 100 with a refill of 100 per minute means a user can fire up to 100 requests instantly if they've been idle, but long term they're pinned to 100 per minute.
Bursty when it's calm, strict when it counts. Chef's kiss.
Are there other algorithms? Sure. Sliding window logs, sliding window counters, leaky buckets, they each solve the boundary problem their own way.
But token bucket hits the sweet spot between "simple enough to actually implement correctly" and "good enough for almost everyone."
For a deeper rabbit hole, the token bucket writeup on Wikipedia is a decent start.
We've got the algorithm. Now, architecturally, where do we run it? Three options, each with a personality.
Which one wins? It genuinely depends.
If you already run an API gateway doing auth, tucking rate limiting in there is a no-brainer.
If you need some exotic custom algorithm, server side gives you room to move.
But for most systems, middleware is the sweet spot: control and operational sanity without soldering the limiter to your business code.
Let's go with middleware and sketch the flow.
We'll store the rules ("premium gets 1000 per hour, free gets 100 per hour") in a configuration service.
The middleware reads those rules, keeps token bucket state in Redis, and makes the call on every incoming request.
When a request lands, the middleware figures out who the user is, pulls their bucket from Redis, and checks for a spare token.
Token available? Decrement it and pass the request along.
Bucket empty? Return a 429
with a Retry-After header so the client knows exactly when to come knocking again.
Be a good host. Tell your guests when the kitchen reopens xD
This is lovely for one rate limiting server. Then, as always, scaling shows up to spoil the party.
Run multiple rate limiter instances against the same Redis and you can hit a classic race condition.
Here's the exact sequence that loses a count:
Both servers read 3. Both decide the request is fine.
Both write 4. We just quietly lost a count, and the counter now lies to us.
Do this enough times under load and your "100 per minute" limit turns into "somewhere around 100, we think, on a good day."
Not exactly the airtight guarantee we promised.
The problem is that read, check, and write are three separate steps, and another server can sneak in between them.
The fix is to make read plus check plus write a single indivisible operation, so nobody can wedge themselves in the middle.
Redis lets us do this with Lua scripts, which run atomically on the server.
The whole check-and-decrement happens as one unit, and the race condition simply cannot occur.
-- KEYS[1] = bucket key
-- ARGV[1] = capacity, ARGV[2] = refill_rate, ARGV[3] = now
local tokens = tonumber(redis.call("GET", KEYS[1]) or ARGV[1])
if tokens >= 1 then
redis.call("DECR", KEYS[1])
return 1 -- allowed
else
return 0 -- rejected, send a 429
end
(That snippet is simplified to keep the point front and center. A production version would compute the refill from elapsed time and store the last-refill timestamp too.)
Atomic operations turn our shaky-under-load counter into something you can actually trust across a fleet of servers.
This is the difference between a rate limiter that works in the demo and one that works on Black Friday.
We covered the load-bearing bits, but a real production rate limiter opens up a bunch of fun follow-up questions worth chewing on:
Each of those is a great whiteboard prompt on its own, and honestly, "fail open vs fail closed" alone has sparked some very heated lunch debates.
So there's our bucket list, completed.
We started with a naive counter, watched it leak requests at window boundaries, upgraded to a token bucket, argued about where to run it, moved the state into Redis so servers could agree, then made the whole thing atomic so scaling couldn't corrupt our counts.
That's the core of nearly every rate limiter you'll ever meet in the wild.
Next time an API hits you with a 429
, you'll know there's a little bucket of tokens somewhere, freshly emptied, quietly telling you to hold your horses.
If you build one, or you have strong opinions on fail open vs fail closed, drop a comment. I'll try to reply before you hit my rate limit.
AI agents write code fast. They also silently remove logic, change behavior, and introduce bugs — without telling you. You often find out in production.
git-lrc fixes this. It hooks into git commit and reviews every diff before it lands. 60-second setup. Completely free.
Any feedback or contributors are welcome! It's online, source-available, and ready for anyone to use.
⭐ Star it on GitHub:
| 🇩🇰 Dansk | 🇪🇸 Español | 🇮🇷 Farsi | 🇫🇮 Suomi | 🇯🇵 日本語 | 🇳🇴 Norsk | 🇵🇹 Português | 🇷🇺 Русский | 🇦🇱 Shqip | 🇨🇳 中文 | 🇮🇳 हिन्दी |
GenAI today is a race car without brakes. It accelerates fast -- you describe something, and large blocks of code appear instantly. But AI agents silently break things: they remove logic, relax constraints, introduce expensive cloud calls, leak credentials, and change behavior -- without telling you. You often find out in production.
** git-lrc is your braking system.** It hooks into
git commit
and runs an AI review on every diff In short, git-lrc helps Prevent Outages, Breaches, and Technical Debt Before They Happen
At a glance: 10 risk categories · 100+ failure patterns tracked · every commit…