{"slug": "too-many-req-a-bucket-list-guide-to-building-a-rate-limiter", "title": "Too Many Req: A Bucket List Guide to Building a Rate Limiter", "summary": "Developer Maneshwar is building git-lrc, a free and source-available Micro AI code reviewer that runs on every commit. In a technical blog post, he provides a guide to designing a rate limiter, covering fixed window counting, the pitfalls of database and in-memory counters, and the token bucket algorithm used by AWS and Stripe.", "body_md": "*Hello, I'm Maneshwar. I'm building git-lrc, a Micro AI code reviewer that runs on every commit. It is free and source-available on Github. Star git-lrc to help devs discover the project. Do give it a try and share your feedback.*\n\nEvery serious API will eventually tell you to sit down and be quiet.\n\nHammer GitHub, Stripe, or AWS a little too eagerly and your requests start bouncing back with a polite but firm `429`\n\n.\n\nI always found that fascinating, so let's build the thing that says no.\n\nBy the end of this post we'll have designed a rate limiter that actually holds up when you put it in front of real traffic, and I promise to only make a reasonable number of bucket puns along the way.\n\nA rate limiter does one job: it decides how many requests a client is allowed to make in a given window of time.\n\nIt protects your system from getting flattened, and it keeps one greedy user from eating everyone else's lunch.\n\nSimple idea. Surprisingly spicy implementation.\n\nLet's build it up piece by piece, the way you'd actually reason through it in an interview or a design doc.\n\nBefore writing a single line, let's agree on what \"good\" looks like.\n\nHere's my wishlist:\n\n`429 Too Many Requests`\n\nCool. Now let's start naive and let reality punch us in the face a few times.\n\nThe simplest thing that could possibly work is **fixed window counting**. Chop time into neat one-minute slices.\n\nGive each user a counter.\n\nEvery request bumps the counter by one.\n\nHit the limit, get rejected, and the counter resets when the next window starts.\n\n```\n# user makes a request\ncount = get(user_id)          # how many so far this minute?\nif count >= 100:\n    reject()                  # 429, come back later\nelse:\n    increment(user_id)\n    allow()\n```\n\nClean. Readable. You could explain it to a rubber duck. So where do we keep this counter?\n\nYour first instinct might be the database.\n\nPlease don't.\n\nWe'd be adding a write to the database on **every request**, which means the thing we built to protect our system is now quietly overloading it.\n\nThat is peak \"I have brought peace, freedom, and a full table scan.\"\n\nOkay, database is out.\n\nWhat about keeping counters **in memory on the server**? Blazing fast. Love it.\n\nExcept it only works if you have exactly one server, and nobody runs one server.\n\nThe moment you scale out, each box keeps its own private counter.\n\nA sneaky user sends 100 requests to Server A and 100 to Server B and walks away with 200 requests per minute while your limit says 100. Whoops.\n\nWhat we actually want is somewhere that is memory-fast *and* shared across every server.\n\nThat's [Redis](https://redis.io/). It's an in-memory data store, it hands us atomic counter primitives like `INCR`\n\n, and it can expire keys automatically so windows reset on their own.\n\nThis is why Redis shows up in basically every rate limiter design ever drawn on a whiteboard.\n\nSo far so good. Now let me ruin it.\n\nFixed windows have a nasty edge case hiding right at the seams.\n\nPicture a limit of 100 requests per minute.\n\nA user fires 100 requests in the *last* 10 seconds of one minute, then another 100 in the *first* 10 seconds of the next minute.\n\nEach window is technically within the limit. Both are 100 or under.\n\nBut zoom out and you'll see 200 requests in a 20 second span, which is very much not the spirit of \"100 per minute.\"\n\nThis happens at every window boundary, and once someone notices the pattern, they will absolutely abuse it.\n\nThe counter has no memory across the boundary, so it cannot see the burst spanning two windows.\n\nWe need an algorithm that thinks in terms of a smooth rate rather than hard resets.\n\nEnter the **token bucket**, the algorithm quietly powering the limits at places like [AWS](https://docs.aws.amazon.com/) and [Stripe](https://docs.stripe.com/rate-limits).\n\nHere's the mental model, and yes, it is literally a bucket.\n\nImagine a bucket that holds tokens. Tokens drip in at a steady rate. Every request has to grab one token to pass. No tokens left? Request gets rejected. That's it.\n\n```\n# refill based on time passed since we last looked\nelapsed = now - last_refill\ntokens  = min(capacity, tokens + elapsed * refill_rate)\nlast_refill = now\n\nif tokens >= 1:\n    tokens -= 1\n    allow()\nelse:\n    reject()   # 429, and tell them when to retry\n```\n\nHere's the diagram version of that decision:\n\nWatch how this fixes our boundary nightmare.\n\nSet the bucket capacity to 100 and the refill rate to 100 per minute.\n\nDuring quiet stretches, tokens pile up toward the cap.\n\nWhen a burst comes in, the user spends whatever tokens they've saved, but they can never outrun the refill rate over the long haul.\n\nNo matter how they time things around a boundary, they cannot conjure tokens that were never added.\n\nThat's the whole trick, and it's genuinely elegant. Two knobs control everything:\n\nCapacity 100 with a refill of 100 per minute means a user can fire up to 100 requests instantly if they've been idle, but long term they're pinned to 100 per minute.\n\nBursty when it's calm, strict when it counts. Chef's kiss.\n\nAre there other algorithms? Sure. Sliding window logs, sliding window counters, leaky buckets, they each solve the boundary problem their own way.\n\nBut token bucket hits the sweet spot between \"simple enough to actually implement correctly\" and \"good enough for almost everyone.\"\n\nFor a deeper rabbit hole, the [token bucket writeup on Wikipedia](https://en.wikipedia.org/wiki/Token_bucket) is a decent start.\n\nWe've got the algorithm. Now, architecturally, where do we run it? Three options, each with a personality.\n\nWhich one wins? It genuinely depends.\n\nIf you already run an API gateway doing auth, tucking rate limiting in there is a no-brainer.\n\nIf you need some exotic custom algorithm, server side gives you room to move.\n\nBut for most systems, **middleware** is the sweet spot: control and operational sanity without soldering the limiter to your business code.\n\nLet's go with middleware and sketch the flow.\n\nWe'll store the rules (\"premium gets 1000 per hour, free gets 100 per hour\") in a configuration service.\n\nThe middleware reads those rules, keeps token bucket state in Redis, and makes the call on every incoming request.\n\nWhen a request lands, the middleware figures out who the user is, pulls their bucket from Redis, and checks for a spare token.\n\nToken available? Decrement it and pass the request along.\n\nBucket empty? Return a `429`\n\nwith a [ Retry-After](https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/Retry-After) header so the client knows exactly when to come knocking again.\n\nBe a good host. Tell your guests when the kitchen reopens xD\n\nThis is lovely for one rate limiting server. Then, as always, scaling shows up to spoil the party.\n\nRun multiple rate limiter instances against the same Redis and you can hit a classic **race condition**.\n\nHere's the exact sequence that loses a count:\n\nBoth servers read 3. Both decide the request is fine.\n\nBoth write 4. We just quietly lost a count, and the counter now lies to us.\n\nDo this enough times under load and your \"100 per minute\" limit turns into \"somewhere around 100, we think, on a good day.\"\n\nNot exactly the airtight guarantee we promised.\n\nThe problem is that read, check, and write are three separate steps, and another server can sneak in between them.\n\nThe fix is to make read plus check plus write a single indivisible operation, so nobody can wedge themselves in the middle.\n\nRedis lets us do this with [Lua scripts](https://redis.io/docs/latest/develop/interact/programmability/eval-intro/), which run atomically on the server.\n\nThe whole check-and-decrement happens as one unit, and the race condition simply cannot occur.\n\n```\n-- KEYS[1] = bucket key\n-- ARGV[1] = capacity, ARGV[2] = refill_rate, ARGV[3] = now\nlocal tokens = tonumber(redis.call(\"GET\", KEYS[1]) or ARGV[1])\nif tokens >= 1 then\n    redis.call(\"DECR\", KEYS[1])\n    return 1        -- allowed\nelse\n    return 0        -- rejected, send a 429\nend\n```\n\n(That snippet is simplified to keep the point front and center. A production version would compute the refill from elapsed time and store the last-refill timestamp too.)\n\nAtomic operations turn our shaky-under-load counter into something you can actually trust across a fleet of servers.\n\nThis is the difference between a rate limiter that works in the demo and one that works on Black Friday.\n\nWe covered the load-bearing bits, but a real production rate limiter opens up a bunch of fun follow-up questions worth chewing on:\n\nEach of those is a great whiteboard prompt on its own, and honestly, \"fail open vs fail closed\" alone has sparked some very heated lunch debates.\n\nSo there's our bucket list, completed.\n\nWe started with a naive counter, watched it leak requests at window boundaries, upgraded to a token bucket, argued about where to run it, moved the state into Redis so servers could agree, then made the whole thing atomic so scaling couldn't corrupt our counts.\n\nThat's the core of nearly every rate limiter you'll ever meet in the wild.\n\nNext time an API hits you with a `429`\n\n, you'll know there's a little bucket of tokens somewhere, freshly emptied, quietly telling you to hold your horses.\n\nIf you build one, or you have strong opinions on fail open vs fail closed, drop a comment. I'll try to reply before you hit my rate limit.\n\nAI agents write code fast. They also silently remove logic, change behavior, and introduce bugs — without telling you. You often find out in production.\n\ngit-lrc fixes this. It hooks into git commit and reviews every diff before it lands. 60-second setup. Completely free.\n\nAny feedback or contributors are welcome! It's online, source-available, and ready for anyone to use.\n\n⭐ Star it on GitHub:\n\n| [🇩🇰 Dansk](https://github.com/HexmosTech/git-lrc/readme/README.da.md) | [🇪🇸 Español](https://github.com/HexmosTech/git-lrc/readme/README.es.md) | [🇮🇷 Farsi](https://github.com/HexmosTech/git-lrc/readme/README.fa.md) | [🇫🇮 Suomi](https://github.com/HexmosTech/git-lrc/readme/README.fi.md) | [🇯🇵 日本語](https://github.com/HexmosTech/git-lrc/readme/README.ja.md) | [🇳🇴 Norsk](https://github.com/HexmosTech/git-lrc/readme/README.nn.md) | [🇵🇹 Português](https://github.com/HexmosTech/git-lrc/readme/README.pt.md) | [🇷🇺 Русский](https://github.com/HexmosTech/git-lrc/readme/README.ru.md) | [🇦🇱 Shqip](https://github.com/HexmosTech/git-lrc/readme/README.sq.md) | [🇨🇳 中文](https://github.com/HexmosTech/git-lrc/readme/README.zh.md) | [🇮🇳 हिन्दी](https://github.com/HexmosTech/git-lrc/readme/README.hi.md) |\n\nGenAI today is a **race car without brakes**. It accelerates fast -- you describe something, and large blocks of code appear instantly. But AI agents *silently break things*: they remove logic, relax constraints, introduce expensive cloud calls, leak credentials, and change behavior -- without telling you. You often find out in production.\n\n** git-lrc is your braking system.** It hooks into\n\n`git commit`\n\nand runs an AI review on every diff In short, git-lrc helps **Prevent Outages, Breaches, and Technical Debt Before They Happen**\n\n**At a glance:** [10 risk categories](https://github.com/HexmosTech/git-lrc#what-git-lrc-checks-for) · [100+ failure patterns tracked](https://github.com/HexmosTech/git-lrc#what-git-lrc-checks-for) · every commit…", "url": "https://wpnews.pro/news/too-many-req-a-bucket-list-guide-to-building-a-rate-limiter", "canonical_source": "https://dev.to/lovestaco/too-many-req-a-bucket-list-guide-to-building-a-rate-limiter-529a", "published_at": "2026-08-23 18:27:12+00:00", "updated_at": "2026-08-23 18:43:12.291112+00:00", "lang": "en", "topics": ["developer-tools", "artificial-intelligence"], "entities": ["Maneshwar", "git-lrc", "GitHub", "Stripe", "AWS", "Redis"], "alternates": {"html": "https://wpnews.pro/news/too-many-req-a-bucket-list-guide-to-building-a-rate-limiter", "markdown": "https://wpnews.pro/news/too-many-req-a-bucket-list-guide-to-building-a-rate-limiter.md", "text": "https://wpnews.pro/news/too-many-req-a-bucket-list-guide-to-building-a-rate-limiter.txt", "jsonld": "https://wpnews.pro/news/too-many-req-a-bucket-list-guide-to-building-a-rate-limiter.jsonld"}}