Carnival time has begun. 🎉
We are organizing a house party, and if you are in and around Bangalore, you are invited. Just one condition:
Don't be late.
We all have that one friend who is always late. We still invite them. Then we start checking the phone.
"Any minute now."
Meanwhile, everyone is standing around, plans are on hold, food is getting cold, ice is melting, and patience is slowly disappearing. At some point, the problem isn't the late friend anymore.
The problem is that everyone keeps waiting for the late friend.
And surprisingly, distributed systems have the exact same problem. That includes your Agentic Applications. 🤖
A service becomes slow or starts failing, and instead of moving on, the rest of the system keeps waiting for it.
Eventually, a problem in one service starts affecting perfectly healthy services. This is when the Circuit Breaker Pattern comes in.
A Circuit Breaker is literally what it sounds like. If you don't know what one does, let an Electrical Engineer explain it to you. 😎
After barely surviving my B.Tech in Electrical Engineering, I was genuinely terrified the first time I saw Circuit Breaker written as a design pattern.
My brain immediately jumped to power systems, transformers, current, voltage, and those end-semester questions that looked like they were personally designed to fail me. 💀
But here's the funny part: software circuit breakers borrow the same idea we learn in electrical engineering. There are three important states:
Closed, Open, and Half-Open
"Half-Open" isn't really a standard electrical state. For our software analogy, think of it as a testing state. After a fault, instead of allowing everything back immediately, we cautiously allow some flow to check whether the system has recovered. If things are fine, we close the circuit. If the fault is still there, we open it again.
With that analogy in mind, let's move into distributed systems.
It's carnival night, so naturally we need pizza. 🍕
Let's say we have a PartyService
which needs to call a PizzaService
to place the order. Normally, everything looks beautiful:
Host
│
▼
PartyService
│
▼
PizzaService
│
▼
Pizza on the way ✅
The host places an order. PizzaService
accepts it. PartyService
gets the confirmation. Everyone eats happily.
But what happens when PizzaService
starts having a bad day?
Now the flow looks like this:
PartyService
│
▼
PizzaService
│
▼
Request Timeout ❌
One order waiting for a few seconds isn't necessarily a problem. But imagine thousands of hungry guests ordering at once:
1000 Orders
│
▼
Slow PizzaService
│
▼
Threads Waiting
│
▼
Connections Waiting
│
▼
Queues Growing
│
▼
Resources Consumed
And now the interesting part begins. PizzaService
is unhealthy. But ** PartyService can become unhealthy too**.
Imagine the following architecture:
Host
│
PartyService
╱ ╲
PizzaService DrinkService
Now PizzaService
goes down. PartyService
keeps calling it. Orders keep waiting. More guests arrive. More threads get occupied. And suddenly PartyService
starts running out of resources too.
PizzaService ❌
│
▼
PartyService ❌
│
▼
Whole Application ❌
One service failed, but the failure travelled through the system. This is what we call a cascading failure — and it's exactly what a Circuit Breaker tries to prevent.
The Circuit Breaker sits between your service and the remote dependency:
PartyService → Circuit Breaker (watches every call) → PizzaService
And that gives us our three states.
The normal state. Requests pass through, and the breaker keeps monitoring.
Request 1 → ✅
Request 2 → ✅
Request 3 → ❌ (network fail, packet drop, kitchens have bad days)
Request 4 → ✅
A few failures don't mean the service is dead. So the circuit stays Closed while the failure rate stays acceptable.
Now failures start piling up:
Request 1 → ❌
Request 2 → ❌
Request 3 → ❌
Request 4 → ❌
At some point the breaker decides: "Yeah... something is wrong here." 😅
CLOSED --(too many failures)--> OPEN
Now it stops sending requests to PizzaService
entirely.
Request → Circuit Breaker → ❌ → Fail Fast / Fallback
The request fails immediately. No unnecessary network call, no waiting for a timeout, no extra load on an already unhealthy service.
But we can't keep the circuit Open forever. What if PizzaService
recovered? After waiting a bit, the breaker moves to Half-Open and lets a few test requests through.
Test Request 1 → ✅
Test Request 2 → ✅ → back to CLOSED 🟢
Test Request 1 → ❌ → back to OPEN 🔴
This is why Half-Open matters. We don't blindly trust the service again — we test it first.
CLOSED 🟢
│
│ too many failures
▼
OPEN 🔴
│
│ wait timeout elapses
▼
HALF-OPEN 🟡
│ │
success failure
│ │
▼ ▼
CLOSED 🟢 OPEN 🔴
The mental model is simple:
Closed → Let requests through
Open → Stop calling the dependency
Half-Open → Carefully test whether it has recovered
Now let's write one.
Using a library without understanding what's happening underneath is a little like driving a car without knowing what the brake does. It works... until it doesn't. 😅
First, our state:
public enum CircuitState {
CLOSED,
OPEN,
HALF_OPEN
}
Now the breaker itself. We track the current state, the failure count, a threshold, how long to stay Open, and when it opened.
import java.util.concurrent.Callable;
import java.util.concurrent.atomic.AtomicInteger;
import java.util.concurrent.atomic.AtomicReference;
public class CircuitBreaker {
private final int failureThreshold;
private final long openWaitMillis;
private final AtomicReference<CircuitState> state =
new AtomicReference<>(CircuitState.CLOSED);
private final AtomicInteger failureCount = new AtomicInteger(0);
private volatile long openedAt = 0;
public CircuitBreaker(int failureThreshold, long openWaitMillis) {
this.failureThreshold = failureThreshold;
this.openWaitMillis = openWaitMillis;
}
public <T> T execute(Callable<T> call, Callable<T> fallback) throws Exception {
if (state.get() == CircuitState.OPEN) {
if (System.currentTimeMillis() - openedAt >= openWaitMillis) {
state.set(CircuitState.HALF_OPEN);
} else {
return fallback.call(); // fail fast
}
}
try {
T result = call.call();
onSuccess();
return result;
} catch (Exception e) {
onFailure();
return fallback.call();
}
}
private void onSuccess() {
failureCount.set(0);
state.set(CircuitState.CLOSED);
}
private void onFailure() {
int failures = failureCount.incrementAndGet();
if (failures >= failureThreshold) {
state.set(CircuitState.OPEN);
openedAt = System.currentTimeMillis();
}
}
}
Notice what happens when the circuit is Open: we don't call the remote service at all. We simply fail fast.
If we configure new CircuitBreaker(3, 10_000)
:
3 failures → OPEN
Wait 10s → HALF_OPEN
Test succeeds → CLOSED
Test fails → back to OPEN
Using it looks like this:
PizzaOrder order = breaker.execute(
() -> pizzaService.order(guestId),
() -> PizzaOrder.fallback(guestId)
);
Once the threshold is hit, the next order doesn't even reach PizzaService
— it fails fast. The dependency is unhealthy, so we stop feeding it traffic.
Before someone copies that class into a production service... please don't. 😅
It's just enough to understand the concept. Real circuit breakers deal with failure rates, sliding windows, concurrency, timeouts, slow calls, exception filtering, metrics, and fallbacks.
For example, is 50 failures bad?
1000 successful requests + 50 failures → probably fine
Last 100 calls: 50 failures (50%) → definitely not fine
A recent window of calls is far more meaningful than a raw count. That's why production apps usually reach for a library.
You might be thinking: "Why not just retry when a request fails?"
Retries solve a different problem.
Imagine PizzaService
is already overloaded, you have 10,000 orders, and every order retries three times. Now you're hammering a struggling service with thousands of extra requests. That's basically knocking on your late friend's door four times instead of once. 😂
So distributed systems combine these patterns, each with a different job:
| Pattern | Question it answers |
|---|---|
| Timeout | How long am I willing to wait? |
| Retry | Should I try again? |
| Circuit Breaker | Should I stop calling this service? |
Here's the part most people miss.
Even I missed this a few weeks back, and it's the real reason this blog exists.
I was building a model orchestrator — a system where one incoming request doesn't just hit a single LLM, it passes through a small pipeline of LLM calls before a final answer comes back:
Request
│
▼
Router LLM → figures out what the user actually wants
│
▼
Agent LLM → does the actual task (search, generate, calculate, etc.)
│
▼
Judge LLM → double-checks the agent's answer before sending it back
│
▼
Final Response
Think of it like a restaurant order going through three people: someone takes your order, someone cooks it, and someone plates it before it reaches your table. If any one of them quietly does nothing and just passes an empty plate along, you still get something — just not what you asked for.
That's basically what happened.
Users started getting weird, inconsistent responses. For the first 30 minutes, I was not worried enough to inspect — the monitoring showed 200 OK for 92% of requests. In most systems, a 200 status means "this worked." So I assumed everything was fine.
It wasn't. When users kept reporting bad answers, we finally looked past the status codes and into the actual response payloads — and found a field I hadn't been watching closely: confidence = 0
.
Here's what was actually happening under the hood:
Router LLM ──fails/times out──▶ falls back silently
Agent LLM ──fails/times out──▶ falls back silently
Judge LLM ──fails/times out──▶ falls back silently
│
▼
Response: 200 OK, confidence = 0
(looks healthy in the logs — nothing "crashed")
Every single stage of the pipeline was failing and quietly falling back to a default response. No exception. No 500 error. No alert. Just a technically-successful HTTP response carrying a useless answer.
I had built in fallback handling for the case where one LLM call in the chain might fail. What I hadn't considered was the entire chain failing together — because the actual root cause wasn't in my code at all. Around the same time, we started seeing 429 Too Many Requests
errors too, and eventually traced it back to a provider-side incident: Anthropic's status page confirmed the model API itself was degraded that day.
The fix was to add a Circuit Breaker at the controller level — the layer that sits in front of all three LLM calls. Before trusting the pipeline with real traffic, the breaker sends a tiny, cheap probe request (just a single token) to confirm the LLM is actually accepting and responding to calls. If the probe fails, the breaker opens immediately and the whole pipeline fails fast with a clear error, instead of quietly returning broken answers dressed up as successes.
That's the real danger with multi-agent systems: a failure doesn't always look like a crash. Sometimes it looks like three fallbacks politely agreeing to lie to you.
GenAI is the new fire. Everyone's building on top of an LLM provider — OpenAI, Anthropic, Gemini, a self-hosted model, whatever. And every one of those calls is... a network call to a remote dependency that can be slow, rate-limited, or down.
Sound familiar?
An LLM call is basically the ultimate late friend:
429 Too Many Requests
when you hit rate limits.Now picture the classic naive setup:
Your AI App → prompt fails or times out → Retry ×3 → Retry ×3 → already rate-limited LLM provider
Retrying a rate-limited model is like ordering more pizza from a kitchen that's already on fire. Every retry pushes you deeper into the rate limit, your latency explodes, and your token bill quietly grows.
A Circuit Breaker in front of the model client fixes this:
AI Request
│
▼
Circuit Breaker
╱ │ ╲
CLOSED OPEN HALF-OPEN
│ │ │
▼ ▼ ▼
LLM Fallback Test call
Provider to LLM
And the fallback is where GenAI apps get to be clever:
Model unavailable
├── Return a cached / similar answer
├── Drop to a smaller, cheaper model
├── Return a "please try again shortly" message
└── Queue the request for later processing
So the next time your agent loops and calls the model 40 times in 5 seconds, remember: even the fanciest AI needs a circuit breaker before it keeps knocking on the provider's door.
Same old pattern, brand new fire.
A fallback doesn't mean return null
. Please don't.
Dependency Down
├── Return cached data
├── Queue the request
├── Drop to a cheaper path
└── Return a graceful error
The important rule: Don't fake success. If the pizza order failed, don't tell the guest "Order Confirmed ✅" just because your fallback needs to return something.
A graceful failure is much better than a successful lie.
Should every exception open the circuit? No.
A 400 Bad Request
(the guest sent garbage input) doesn't mean the service is unhealthy. But these are strong signals that it is:
Connection refused
Timeout
503 Service Unavailable
429 Too Many Requests
So when configuring a breaker, one of the most important questions is:
Which failures should actually count?
That decision is application-specific.
That's the Circuit Breaker.
Whenever you hear Circuit Breaker, remember the three states:
CLOSED → requests flow normally
OPEN → requests stop immediately
HALF-OPEN → a few requests test recovery
Or, in Bangalore house-party terms:
Friend is coming → Everyone waits → Friend keeps failing → STOP WAITING → Party Continues 😎
A Circuit Breaker doesn't fix the failing service. It protects the rest of your system while that service is failing — whether that's a pizza shop, a payment API, or your favourite LLM.
A failure is inevitable. A cascading failure doesn't have to be.
Firstly, thanks a lot if you're still reading this far. 🎉
Remember, distributed systems are a bit like that one late friend — the failures are inevitable, but how you handle them is entirely up to you. A circuit breaker won't fix a broken service, but it'll stop one bad dependency from taking your whole system (or your whole party) down with it.
The purpose of this blog was to demystify a pattern that sounds intimidating but is genuinely simple once you see it as three states and a timer. Whether it's a pizza order, a payment API, or an LLM call — the same rule applies: stop waiting, fail fast, and test recovery before trusting again.
The more you work with distributed systems, the more you realise the most useful ideas are simple:
And if the answer is no... open the circuit.
I plan to write more on distributed systems patterns and backend engineering — the kind of stuff that shows up in real production systems, not just textbooks.
Till then, go build something that doesn't wait around for late friends. 😄
If you run an organisation and want me to write or create video tutorials, please do connect with me 🤝