Why Your LLM Calls Need a Circuit Breaker: A Lesson from Last Party 🔌

A developer explains how the circuit breaker pattern, borrowed from electrical engineering, can prevent cascading failures in agentic applications and LLM-based systems. The pattern monitors remote service calls and opens the circuit when failure rates exceed a threshold, protecting healthy services from being dragged down by a failing dependency.

Carnival time has begun. 🎉 We are organizing a house party, and if you are in and around Bangalore, you are invited. Just one condition: Don't be late. We all have that one friend who is always late. We still invite them. Then we start checking the phone. "Any minute now." Meanwhile, everyone is standing around, plans are on hold, food is getting cold, ice is melting, and patience is slowly disappearing. At some point, the problem isn't the late friend anymore. The problem is that everyone keeps waiting for the late friend. And surprisingly, distributed systems have the exact same problem. That includes your Agentic Applications . 🤖 A service becomes slow or starts failing, and instead of moving on, the rest of the system keeps waiting for it. Eventually, a problem in one service starts affecting perfectly healthy services. This is when the Circuit Breaker Pattern comes in. A Circuit Breaker is literally what it sounds like. If you don't know what one does, let an Electrical Engineer explain it to you. 😎 After barely surviving my B.Tech in Electrical Engineering, I was genuinely terrified the first time I saw Circuit Breaker written as a design pattern . My brain immediately jumped to power systems, transformers, current, voltage, and those end-semester questions that looked like they were personally designed to fail me. 💀 But here's the funny part: software circuit breakers borrow the same idea we learn in electrical engineering. There are three important states: Closed, Open, and Half-Open "Half-Open" isn't really a standard electrical state. For our software analogy, think of it as a testing state . After a fault, instead of allowing everything back immediately, we cautiously allow some flow to check whether the system has recovered. If things are fine, we close the circuit. If the fault is still there, we open it again. With that analogy in mind, let's move into distributed systems. It's carnival night, so naturally we need pizza. 🍕 Let's say we have a PartyService which needs to call a PizzaService to place the order. Normally, everything looks beautiful: Host │ ▼ PartyService │ ▼ PizzaService │ ▼ Pizza on the way ✅ The host places an order. PizzaService accepts it. PartyService gets the confirmation. Everyone eats happily. But what happens when PizzaService starts having a bad day? Now the flow looks like this: PartyService │ ▼ PizzaService │ ▼ Request Timeout ❌ One order waiting for a few seconds isn't necessarily a problem. But imagine thousands of hungry guests ordering at once: 1000 Orders │ ▼ Slow PizzaService │ ▼ Threads Waiting │ ▼ Connections Waiting │ ▼ Queues Growing │ ▼ Resources Consumed And now the interesting part begins. PizzaService is unhealthy. But PartyService can become unhealthy too . Imagine the following architecture: Host │ PartyService ╱ ╲ PizzaService DrinkService Now PizzaService goes down. PartyService keeps calling it. Orders keep waiting. More guests arrive. More threads get occupied. And suddenly PartyService starts running out of resources too. PizzaService ❌ │ ▼ PartyService ❌ │ ▼ Whole Application ❌ One service failed, but the failure travelled through the system. This is what we call a cascading failure — and it's exactly what a Circuit Breaker tries to prevent. The Circuit Breaker sits between your service and the remote dependency: PartyService → Circuit Breaker watches every call → PizzaService And that gives us our three states. The normal state. Requests pass through, and the breaker keeps monitoring. Request 1 → ✅ Request 2 → ✅ Request 3 → ❌ network fail, packet drop, kitchens have bad days Request 4 → ✅ A few failures don't mean the service is dead. So the circuit stays Closed while the failure rate stays acceptable. Now failures start piling up: Request 1 → ❌ Request 2 → ❌ Request 3 → ❌ Request 4 → ❌ At some point the breaker decides: "Yeah... something is wrong here." 😅 php CLOSED -- too many failures -- OPEN Now it stops sending requests to PizzaService entirely. Request → Circuit Breaker → ❌ → Fail Fast / Fallback The request fails immediately. No unnecessary network call, no waiting for a timeout, no extra load on an already unhealthy service. But we can't keep the circuit Open forever. What if PizzaService recovered? After waiting a bit, the breaker moves to Half-Open and lets a few test requests through. Test Request 1 → ✅ Test Request 2 → ✅ → back to CLOSED 🟢 Test Request 1 → ❌ → back to OPEN 🔴 This is why Half-Open matters. We don't blindly trust the service again — we test it first . CLOSED 🟢 │ │ too many failures ▼ OPEN 🔴 │ │ wait timeout elapses ▼ HALF-OPEN 🟡 │ │ success failure │ │ ▼ ▼ CLOSED 🟢 OPEN 🔴 The mental model is simple: Closed → Let requests through Open → Stop calling the dependency Half-Open → Carefully test whether it has recovered Now let's write one. Using a library without understanding what's happening underneath is a little like driving a car without knowing what the brake does. It works... until it doesn't. 😅 First, our state: public enum CircuitState { CLOSED, OPEN, HALF OPEN } Now the breaker itself. We track the current state, the failure count, a threshold, how long to stay Open, and when it opened. python import java.util.concurrent.Callable; import java.util.concurrent.atomic.AtomicInteger; import java.util.concurrent.atomic.AtomicReference; public class CircuitBreaker { private final int failureThreshold; private final long openWaitMillis; private final AtomicReference<CircuitState state = new AtomicReference< CircuitState.CLOSED ; private final AtomicInteger failureCount = new AtomicInteger 0 ; private volatile long openedAt = 0; public CircuitBreaker int failureThreshold, long openWaitMillis { this.failureThreshold = failureThreshold; this.openWaitMillis = openWaitMillis; } public <T T execute Callable<T call, Callable<T fallback throws Exception { if state.get == CircuitState.OPEN { if System.currentTimeMillis - openedAt = openWaitMillis { state.set CircuitState.HALF OPEN ; } else { return fallback.call ; // fail fast } } try { T result = call.call ; onSuccess ; return result; } catch Exception e { onFailure ; return fallback.call ; } } private void onSuccess { failureCount.set 0 ; state.set CircuitState.CLOSED ; } private void onFailure { int failures = failureCount.incrementAndGet ; if failures = failureThreshold { state.set CircuitState.OPEN ; openedAt = System.currentTimeMillis ; } } } Notice what happens when the circuit is Open: we don't call the remote service at all. We simply fail fast. If we configure new CircuitBreaker 3, 10 000 : 3 failures → OPEN Wait 10s → HALF OPEN Test succeeds → CLOSED Test fails → back to OPEN Using it looks like this: php PizzaOrder order = breaker.execute - pizzaService.order guestId , - PizzaOrder.fallback guestId ; Once the threshold is hit, the next order doesn't even reach PizzaService — it fails fast. The dependency is unhealthy, so we stop feeding it traffic. Before someone copies that class into a production service... please don't. 😅 It's just enough to understand the concept. Real circuit breakers deal with failure rates, sliding windows, concurrency, timeouts, slow calls, exception filtering, metrics, and fallbacks . For example, is 50 failures bad? 1000 successful requests + 50 failures → probably fine Last 100 calls: 50 failures 50% → definitely not fine A recent window of calls is far more meaningful than a raw count. That's why production apps usually reach for a library. You might be thinking: "Why not just retry when a request fails?" Retries solve a different problem. Imagine PizzaService is already overloaded, you have 10,000 orders, and every order retries three times. Now you're hammering a struggling service with thousands of extra requests. That's basically knocking on your late friend's door four times instead of once. 😂 So distributed systems combine these patterns, each with a different job: | Pattern | Question it answers | |---|---| | Timeout | How long am I willing to wait? | | Retry | Should I try again? | | Circuit Breaker | Should I stop calling this service? | Here's the part most people miss. Even I missed this a few weeks back, and it's the real reason this blog exists. I was building a model orchestrator — a system where one incoming request doesn't just hit a single LLM, it passes through a small pipeline of LLM calls before a final answer comes back: Request │ ▼ Router LLM → figures out what the user actually wants │ ▼ Agent LLM → does the actual task search, generate, calculate, etc. │ ▼ Judge LLM → double-checks the agent's answer before sending it back │ ▼ Final Response Think of it like a restaurant order going through three people: someone takes your order, someone cooks it, and someone plates it before it reaches your table. If any one of them quietly does nothing and just passes an empty plate along, you still get something — just not what you asked for. That's basically what happened. Users started getting weird, inconsistent responses. For the first 30 minutes, I was not worried enough to inspect — the monitoring showed 200 OK for 92% of requests . In most systems, a 200 status means "this worked." So I assumed everything was fine. It wasn't. When users kept reporting bad answers, we finally looked past the status codes and into the actual response payloads — and found a field I hadn't been watching closely: confidence = 0 . Here's what was actually happening under the hood: Router LLM ──fails/times out──▶ falls back silently Agent LLM ──fails/times out──▶ falls back silently Judge LLM ──fails/times out──▶ falls back silently │ ▼ Response: 200 OK, confidence = 0 looks healthy in the logs — nothing "crashed" Every single stage of the pipeline was failing and quietly falling back to a default response. No exception. No 500 error. No alert. Just a technically-successful HTTP response carrying a useless answer. I had built in fallback handling for the case where one LLM call in the chain might fail. What I hadn't considered was the entire chain failing together — because the actual root cause wasn't in my code at all. Around the same time, we started seeing 429 Too Many Requests errors too, and eventually traced it back to a provider-side incident: Anthropic's status page confirmed the model API itself was degraded that day. The fix was to add a Circuit Breaker at the controller level — the layer that sits in front of all three LLM calls. Before trusting the pipeline with real traffic, the breaker sends a tiny, cheap probe request just a single token to confirm the LLM is actually accepting and responding to calls. If the probe fails, the breaker opens immediately and the whole pipeline fails fast with a clear error, instead of quietly returning broken answers dressed up as successes. That's the real danger with multi-agent systems: a failure doesn't always look like a crash. Sometimes it looks like three fallbacks politely agreeing to lie to you. GenAI is the new fire. Everyone's building on top of an LLM provider — OpenAI, Anthropic, Gemini, a self-hosted model, whatever. And every one of those calls is... a network call to a remote dependency that can be slow, rate-limited, or down . Sound familiar? An LLM call is basically the ultimate late friend: 429 Too Many Requests when you hit rate limits.Now picture the classic naive setup: Your AI App → prompt fails or times out → Retry ×3 → Retry ×3 → already rate-limited LLM provider Retrying a rate-limited model is like ordering more pizza from a kitchen that's already on fire. Every retry pushes you deeper into the rate limit, your latency explodes, and your token bill quietly grows. A Circuit Breaker in front of the model client fixes this: AI Request │ ▼ Circuit Breaker ╱ │ ╲ CLOSED OPEN HALF-OPEN │ │ │ ▼ ▼ ▼ LLM Fallback Test call Provider to LLM And the fallback is where GenAI apps get to be clever: Model unavailable ├── Return a cached / similar answer ├── Drop to a smaller, cheaper model ├── Return a "please try again shortly" message └── Queue the request for later processing So the next time your agent loops and calls the model 40 times in 5 seconds, remember: even the fanciest AI needs a circuit breaker before it keeps knocking on the provider's door. Same old pattern, brand new fire. A fallback doesn't mean return null . Please don't. Dependency Down ├── Return cached data ├── Queue the request ├── Drop to a cheaper path └── Return a graceful error The important rule: Don't fake success. If the pizza order failed, don't tell the guest "Order Confirmed ✅" just because your fallback needs to return something . A graceful failure is much better than a successful lie. Should every exception open the circuit? No. A 400 Bad Request the guest sent garbage input doesn't mean the service is unhealthy. But these are strong signals that it is: Connection refused Timeout 503 Service Unavailable 429 Too Many Requests So when configuring a breaker, one of the most important questions is: Which failures should actually count? That decision is application-specific. That's the Circuit Breaker. Whenever you hear Circuit Breaker , remember the three states: CLOSED → requests flow normally OPEN → requests stop immediately HALF-OPEN → a few requests test recovery Or, in Bangalore house-party terms: Friend is coming → Everyone waits → Friend keeps failing → STOP WAITING → Party Continues 😎 A Circuit Breaker doesn't fix the failing service. It protects the rest of your system while that service is failing — whether that's a pizza shop, a payment API, or your favourite LLM. A failure is inevitable. A cascading failure doesn't have to be. Firstly, thanks a lot if you're still reading this far. 🎉 Remember, distributed systems are a bit like that one late friend — the failures are inevitable, but how you handle them is entirely up to you. A circuit breaker won't fix a broken service, but it'll stop one bad dependency from taking your whole system or your whole party down with it. The purpose of this blog was to demystify a pattern that sounds intimidating but is genuinely simple once you see it as three states and a timer. Whether it's a pizza order, a payment API, or an LLM call — the same rule applies: stop waiting, fail fast, and test recovery before trusting again. The more you work with distributed systems, the more you realise the most useful ideas are simple: And if the answer is no... open the circuit . I plan to write more on distributed systems patterns and backend engineering — the kind of stuff that shows up in real production systems, not just textbooks. Till then, go build something that doesn't wait around for late friends. 😄 If you run an organisation and want me to write or create video tutorials, please do connect with me 🤝