Hello, I'm Maneshwar, and I'm building LiveReview — a blast-radius aware AI code review built for your business-critical systems. Star us to help devs discover the project, give it a try, and share your feedback to help improve the product.
Here's a fun fact that nobody puts on the slide at the AI meetup.
Your agent can be perfect. The prompt can be tuned within an inch of its life. The eval score can be a smug 94%.
None of that matters the moment your product gets popular.
One user hits your FastAPI endpoint, the agent runs, the model answers, everybody's happy.
Ten thousand users hit it in the same five minutes, and suddenly the model is the least interesting part of your outage.
The bottleneck moved. It always moves. It moved from "can the model answer this" to "can the system around the model survive being asked."
That system is what this post is about. Seven patterns, one request, and the boring infrastructure work that decides whether your AI product works for a demo or for a Tuesday afternoon in production.
Picture the AI service you already built: FastAPI route, an agent workflow, a vector store, tracing, evals, the whole thing.
That entire box becomes one small rectangle in a bigger picture starting today.
Nothing inside that rectangle changes. FastAPI still handles the route, the agent still calls the model, tracing still tells you what happened.
What's new is everything around it, deciding who gets in, what waits, what fails safely, and how much capacity exists in the first place.
An API gateway sits in front of your application and handles traffic before it ever reaches your route handlers.
It checks who's calling, rejects garbage requests before they cost you compute, tags each one with a request ID, and routes it to the right internal service.
Think of it as reception for a big office building. One door in, one person checking where you need to go, and you never have to know which floor anything is actually on.
One distinction that trips people up: this is not the same as a model gateway. An API gateway manages traffic coming into your app from users. A model gateway manages calls going out to OpenAI, Anthropic, or whoever's serving your model. Same word, opposite direction.
A gateway controlling who can knock on the door still lets in more valid traffic than you can serve. The next question isn't "who's allowed in," it's "how much work do we let start."
Plain rate limiting counts requests per user per minute. Fine for a CRUD API. Not fine for an LLM app, where request count barely correlates with cost.
One call classifies "yes" or "no" in ten tokens. Another retrieves eight documents, calls three tools, and streams back four paragraphs. Same "one request" on your dashboard, wildly different bill.
So AI systems widen the definition to admission control: cap requests, cap input tokens, cap output tokens, cap concurrent model calls, whatever actually predicts cost in your system.
When you're over budget, the honest answer is an HTTP 429 with a Retry-After header, the standard response for telling a client to back off instead of quietly queuing infinite work. That's called back pressure: push the slowdown back to whoever's asking, before it becomes your problem.
A gateway and a rate limiter, and you've still got a pile of accepted requests all asking suspiciously similar questions.
If five hundred people ask about the same public doc today, the first request should do the retrieval and embedding work, and the other 499 should get it basically for free.
Redis in front of embeddings, retrieval results, or common model responses buys you real latency and cost wins.
The catch, and it's a big one for anything touching an LLM: caching an answer is a claim that it's still correct and still safe for whoever's asking.
A private answer generated for one user is not a cache entry, it's a leak, if it ever gets served to somebody else. A cached fact from three months ago about pricing or an API limit is not "reused work," it's a wrong answer with good latency.
Every cache entry needs an expiry, a scope (user, tenant, permission level), and an invalidation path tied to the source changing. Skip any of those three and the cache stops saving you money and starts costing you incidents.
Caching removes duplicate work. It doesn't remove the different work, and there's a lot of AI-adjacent work that genuinely doesn't need to finish before you respond to the user: document ingestion, embedding generation on upload, a long report, an email.
That's what a durable queue is for. Take the order, hand the customer a ticket number, let the kitchen cook at the kitchen's pace instead of the counter's.
A few details that separate "we added a queue" from "we added a queue that actually holds":
Bound the backlog. An unbounded queue doesn't prevent an outage, it just delays it and makes it bigger when it finally lands.
Route failures somewhere visible. A job that keeps failing should land in a dead letter queue for a human to look at, not retry silently forever and quietly eat your worker capacity.
Make retried actions idempotent. At-least-once delivery means a worker can see the same job twice. If "process this job" means "charge this card" or "send this email," processing it twice is a bug wearing a queue's clothing.
Separate lanes for separate work. One giant document-ingestion job parked in the same queue as your live chat's background tasks will happily starve the thing your paying customers are waiting on.
def process_job(job_id, payload):
if redis.setnx(f"processed:{job_id}", 1):
redis.expire(f"processed:{job_id}", 86400)
do_the_actual_work(payload)
Now the harder failure mode. Not "the dependency is down," which fails fast and cleanly, but "the dependency is slow," which is much worse.
A dependency that's fully down refuses your call in milliseconds. Your worker's free again instantly, checkout fails cleanly, the rest of the site keeps serving.
A dependency that's slow holds that same worker for thirty seconds while it decides whether to answer. Do that across your whole pool and a random product page dies for a problem it never even called.
Three tools handle this, in order:
Timeouts. Decide up front how long you're willing to wait. No timeout means every slow dependency gets to set your latency for you.
Retries, with backoff and jitter. A brief retry after a random-ish delay handles transient blips. AWS has the definitive writeup on why the jitter matters: without it, every client backs off on the same clock and retries in a synchronized wave that looks exactly like the outage you were trying to survive.
The circuit breaker, for when it's not transient. Martin Fowler's original writeup frames it exactly like the breaker in your fuse box: too many failures and the circuit trips open, calls stop reaching the broken dependency, and everyone's spared the wait.
Closed is normal operation. Open is refusing everything instantly while the failing dependency recovers off the hook. Half-open is the cautious part: let a handful of test calls through, and only reopen the floodgates if they succeed.
When the circuit's open, don't fill the gap with a guess. Serve a tested fallback model, a verified cached answer, or an honest "this feature's briefly unavailable." Graceful degradation means reduced functionality, never a confident wrong answer.
And keep failing dependencies from sharing a resource pool with healthy ones — separate connection pools per dependency, a pattern usually called a bulkhead, named for the same reason ships have them: one flooded compartment shouldn't sink the whole hull.
Every one of those live requests is still landing on a single instance of your core AI service, and one healthy server still has a ceiling.
Run several copies, put a load balancer in front, and it hands each request to whichever healthy instance has room.
Worth separating from the gateway, since people conflate them: the gateway decides which service a request should reach. The load balancer decides which copy of that service handles it. Some managed products bundle both jobs into one offering, but they're still two different decisions.
The thing that makes running several copies safe is keeping them stateless. Durable conversation history, workflow checkpoints, job status — put those in Postgres. Short-lived shared state goes in something like Redis. Then any instance can pick up any request, and a retried call resumes from a saved checkpoint instead of restarting the whole agent loop from scratch.
Everything above buys time. Eventually, if traffic keeps growing, distributing the load across existing copies stops being enough, and you actually need more of them.
Autoscaling adds or removes instances as demand shifts, so you're not paying for a night's worth of GPU capacity at 3am for zero users.
The part worth getting right is what signal drives it. CPU alone is a bad proxy for a GPU-backed model server, since you can be maxed out on the accelerator while CPU idles. Better signals: queue depth, how long the oldest task's been waiting, active requests, or GPU utilization directly.
New capacity also isn't instant. A stateless API pod might come up in seconds. A model server weights into GPU memory can take minutes. If your traffic pattern is predictable (the daily 9am spike, say), keep some capacity warm ahead of it rather than reacting cold.
This is also why autoscaling comes last in the list, not first. More servers don't fix a cache serving stale answers, a retry storm with no jitter, or a queue with no bound. Control the work first. Add capacity second.
A user's request hits the API gateway, which gives it one controlled entrance and routes it toward the right service.
The rate limiter checks whether this user, and this kind of request, is within budget.
The load balancer picks a healthy instance of the core AI service.
That instance checks the cache first: if the answer is safe to reuse, it comes back immediately.
If the request needs an answer now, it stays on the live path. If it's kicking off longer background work, it drops into the queue for a worker to pick up.
Timeouts and circuit breakers guard every call the instance makes outward, to the model provider, the vector store, any tool. The autoscaler is watching the whole system and adjusting instance count as it goes.
And inside that one small rectangle, nothing about the actual AI work changed: FastAPI handles the route, the agent runs its loop, the model generates the answer, tracing records what happened, evals score whether it was any good.
That last point is the one worth sitting with. Metrics tell you if the system's fast. Traces tell you where a request spent its time. Evals tell you if the answer was actually worth sending.
None of these seven patterns touch that last question, and they were never meant to. Scaling a wrong answer faster is not a win, it's just a faster wrong answer reaching more people.
If you ever get a system design question about this in an interview, resist the urge to start naming seven cloud products. Start with the request: how much traffic, what needs to answer immediately versus what can wait, and which dependency is going to be the first one to have a bad day. Add a pattern only when it's solving a problem you can point to.
The model was never the bug. The system that decides who gets to ask it something, and what happens while it's thinking, always was.
Your team's attention is limited, and the deluge of AI-generated code is making it harder to keep production secure and reliable without slowing you down.
I'm building LiveReview, a blast-radius aware AI code review built for your business-critical systems.
Instead of presenting every diff with equal emphasis, LiveReview scores each change by blast radius — how far its impact reaches through your call graph — so you can focus attention where it actually matters.
Spend code review effort where business risk is highest — not spread evenly across every diff.
⭐ Star it on GitHub:
LiveReview is an AI code reviewer that scores every hunk of a diff by blast radius: how far a change reaches through your call graph, how much persistent state it touches, and how well-tested it is. A 3-line change to a shared auth check can outrank a 300-line UI tweak. Your team's attention goes to the highest-risk code first, not spread evenly across every diff.
LiveReview's Blast Radius & Review Priority scoring, live in the diff viewer.
| The exact math, not a black box | Visualize blast radius at a glance | Every factor that feeds the score |
|---|
Here's the goal:
Click below to try LiveReview with your codebase: