{"slug": "why-your-llm-calls-need-a-circuit-breaker-a-lesson-from-last-party", "title": "Why Your LLM Calls Need a Circuit Breaker: A Lesson from Last Party 🔌", "summary": "A developer explains how the circuit breaker pattern, borrowed from electrical engineering, can prevent cascading failures in agentic applications and LLM-based systems. The pattern monitors remote service calls and opens the circuit when failure rates exceed a threshold, protecting healthy services from being dragged down by a failing dependency.", "body_md": "Carnival time has begun. 🎉\n\nWe are organizing a house party, and if you are in and around Bangalore, you are invited. Just one condition:\n\nDon't be late.\n\nWe all have that *one* friend who is always late. We still invite them. Then we start checking the phone.\n\n\"Any minute now.\"\n\nMeanwhile, everyone is standing around, plans are on hold, food is getting cold, ice is melting, and patience is slowly disappearing. At some point, the problem isn't the late friend anymore.\n\n**The problem is that everyone keeps waiting for the late friend.**\n\nAnd surprisingly, distributed systems have the exact same problem. That includes your **Agentic Applications**. 🤖\n\nA service becomes slow or starts failing, and instead of moving on, the rest of the system keeps waiting for it.\n\nEventually, a problem in **one service** starts affecting perfectly healthy services. This is when the **Circuit Breaker Pattern** comes in.\n\nA Circuit Breaker is literally what it sounds like. If you don't know what one does, let an Electrical Engineer explain it to you. 😎\n\nAfter barely surviving my B.Tech in Electrical Engineering, I was genuinely terrified the first time I saw **Circuit Breaker** written as a **design pattern**.\n\nMy brain immediately jumped to power systems, transformers, current, voltage, and those end-semester questions that looked like they were personally designed to fail me. 💀\n\nBut here's the funny part: software circuit breakers borrow the same idea we learn in electrical engineering. There are three important states:\n\nClosed, Open, and Half-Open\n\n\"Half-Open\" isn't really a standard electrical state. For our software analogy, think of it as a **testing state**. After a fault, instead of allowing everything back immediately, we cautiously allow some flow to check whether the system has recovered. If things are fine, we close the circuit. If the fault is still there, we open it again.\n\nWith that analogy in mind, let's move into distributed systems.\n\nIt's carnival night, so naturally we need pizza. 🍕\n\nLet's say we have a `PartyService`\n\nwhich needs to call a `PizzaService`\n\nto place the order. Normally, everything looks beautiful:\n\n```\nHost\n │\n ▼\nPartyService\n │\n ▼\nPizzaService\n │\n ▼\nPizza on the way ✅\n```\n\nThe host places an order. `PizzaService`\n\naccepts it. `PartyService`\n\ngets the confirmation. Everyone eats happily.\n\nBut what happens when `PizzaService`\n\nstarts having a bad day?\n\nNow the flow looks like this:\n\n```\nPartyService\n │\n ▼\nPizzaService\n │\n ▼\nRequest Timeout ❌\n```\n\nOne order waiting for a few seconds isn't necessarily a problem. But imagine thousands of hungry guests ordering at once:\n\n```\n1000 Orders\n │\n ▼\nSlow PizzaService\n │\n ▼\nThreads Waiting\n │\n ▼\nConnections Waiting\n │\n ▼\nQueues Growing\n │\n ▼\nResources Consumed\n```\n\nAnd now the interesting part begins. `PizzaService`\n\nis unhealthy. But ** PartyService can become unhealthy too**.\n\nImagine the following architecture:\n\n```\n                Host\n                 │\n            PartyService\n              ╱       ╲\n     PizzaService     DrinkService\n```\n\nNow `PizzaService`\n\ngoes down. `PartyService`\n\nkeeps calling it. Orders keep waiting. More guests arrive. More threads get occupied. And suddenly `PartyService`\n\nstarts running out of resources too.\n\n```\nPizzaService ❌\n │\n ▼\nPartyService ❌\n │\n ▼\nWhole Application ❌\n```\n\nOne service failed, but the failure travelled through the system. This is what we call a **cascading failure** — and it's exactly what a Circuit Breaker tries to prevent.\n\nThe Circuit Breaker sits between your service and the remote dependency:\n\n```\nPartyService → Circuit Breaker (watches every call) → PizzaService\n```\n\nAnd that gives us our three states.\n\nThe normal state. Requests pass through, and the breaker keeps monitoring.\n\n```\nRequest 1 → ✅\nRequest 2 → ✅\nRequest 3 → ❌ (network fail, packet drop, kitchens have bad days)\nRequest 4 → ✅\n```\n\nA few failures don't mean the service is dead. So the circuit stays **Closed** while the failure rate stays acceptable.\n\nNow failures start piling up:\n\n```\nRequest 1 → ❌\nRequest 2 → ❌\nRequest 3 → ❌\nRequest 4 → ❌\n```\n\nAt some point the breaker decides: *\"Yeah... something is wrong here.\"* 😅\n\n``` php\nCLOSED --(too many failures)--> OPEN\n```\n\nNow it **stops sending requests** to `PizzaService`\n\nentirely.\n\n```\nRequest → Circuit Breaker → ❌ → Fail Fast / Fallback\n```\n\nThe request fails immediately. No unnecessary network call, no waiting for a timeout, no extra load on an already unhealthy service.\n\nBut we can't keep the circuit Open forever. What if `PizzaService`\n\nrecovered? After waiting a bit, the breaker moves to **Half-Open** and lets a few test requests through.\n\n```\nTest Request 1 → ✅\nTest Request 2 → ✅ → back to CLOSED 🟢\n\nTest Request 1 → ❌ → back to OPEN 🔴\n```\n\nThis is why Half-Open matters. We don't blindly trust the service again — **we test it first**.\n\n```\n        CLOSED 🟢\n           │\n           │  too many failures\n           ▼\n         OPEN 🔴\n           │\n           │  wait timeout elapses\n           ▼\n      HALF-OPEN 🟡\n       │       │\n  success    failure\n       │       │\n       ▼       ▼\n   CLOSED 🟢  OPEN 🔴\n```\n\nThe mental model is simple:\n\nClosed → Let requests through\n\nOpen → Stop calling the dependency\n\nHalf-Open → Carefully test whether it has recovered\n\nNow let's write one.\n\nUsing a library without understanding what's happening underneath is a little like driving a car without knowing what the brake does. It works... until it doesn't. 😅\n\nFirst, our state:\n\n```\npublic enum CircuitState {\n    CLOSED,\n    OPEN,\n    HALF_OPEN\n}\n```\n\nNow the breaker itself. We track the current state, the failure count, a threshold, how long to stay Open, and when it opened.\n\n``` python\nimport java.util.concurrent.Callable;\nimport java.util.concurrent.atomic.AtomicInteger;\nimport java.util.concurrent.atomic.AtomicReference;\n\npublic class CircuitBreaker {\n\n    private final int failureThreshold;\n    private final long openWaitMillis;\n\n    private final AtomicReference<CircuitState> state =\n            new AtomicReference<>(CircuitState.CLOSED);\n    private final AtomicInteger failureCount = new AtomicInteger(0);\n    private volatile long openedAt = 0;\n\n    public CircuitBreaker(int failureThreshold, long openWaitMillis) {\n        this.failureThreshold = failureThreshold;\n        this.openWaitMillis = openWaitMillis;\n    }\n\n    public <T> T execute(Callable<T> call, Callable<T> fallback) throws Exception {\n        if (state.get() == CircuitState.OPEN) {\n            if (System.currentTimeMillis() - openedAt >= openWaitMillis) {\n                state.set(CircuitState.HALF_OPEN);\n            } else {\n                return fallback.call(); // fail fast\n            }\n        }\n\n        try {\n            T result = call.call();\n            onSuccess();\n            return result;\n        } catch (Exception e) {\n            onFailure();\n            return fallback.call();\n        }\n    }\n\n    private void onSuccess() {\n        failureCount.set(0);\n        state.set(CircuitState.CLOSED);\n    }\n\n    private void onFailure() {\n        int failures = failureCount.incrementAndGet();\n        if (failures >= failureThreshold) {\n            state.set(CircuitState.OPEN);\n            openedAt = System.currentTimeMillis();\n        }\n    }\n}\n```\n\nNotice what happens when the circuit is Open: we don't call the remote service at all. We simply fail fast.\n\nIf we configure `new CircuitBreaker(3, 10_000)`\n\n:\n\n```\n3 failures       → OPEN\nWait 10s         → HALF_OPEN\nTest succeeds    → CLOSED\nTest fails       → back to OPEN\n```\n\nUsing it looks like this:\n\n``` php\nPizzaOrder order = breaker.execute(\n    () -> pizzaService.order(guestId),\n    () -> PizzaOrder.fallback(guestId)\n);\n```\n\nOnce the threshold is hit, the next order doesn't even reach `PizzaService`\n\n— it fails fast. **The dependency is unhealthy, so we stop feeding it traffic.**\n\nBefore someone copies that class into a production service... please don't. 😅\n\nIt's just enough to understand the concept. Real circuit breakers deal with **failure rates, sliding windows, concurrency, timeouts, slow calls, exception filtering, metrics, and fallbacks**.\n\nFor example, is 50 failures bad?\n\n```\n1000 successful requests + 50 failures  → probably fine\nLast 100 calls: 50 failures (50%)       → definitely not fine\n```\n\nA **recent window of calls** is far more meaningful than a raw count. That's why production apps usually reach for a library.\n\nYou might be thinking: *\"Why not just retry when a request fails?\"*\n\nRetries solve a different problem.\n\nImagine `PizzaService`\n\nis already overloaded, you have 10,000 orders, and every order retries three times. Now you're hammering a struggling service with thousands of *extra* requests. That's basically knocking on your late friend's door **four times** instead of once. 😂\n\nSo distributed systems combine these patterns, each with a different job:\n\n| Pattern | Question it answers |\n|---|---|\n| Timeout | How long am I willing to wait? |\n| Retry | Should I try again? |\n| Circuit Breaker | Should I stop calling this service? |\n\nHere's the part most people miss.\n\nEven I missed this a few weeks back, and it's the real reason this blog exists.\n\nI was building a **model orchestrator** — a system where one incoming request doesn't just hit a single LLM, it passes through a small pipeline of LLM calls before a final answer comes back:\n\n```\nRequest\n   │\n   ▼\nRouter LLM   → figures out what the user actually wants\n   │\n   ▼\nAgent LLM    → does the actual task (search, generate, calculate, etc.)\n   │\n   ▼\nJudge LLM    → double-checks the agent's answer before sending it back\n   │\n   ▼\nFinal Response\n```\n\nThink of it like a restaurant order going through three people: someone takes your order, someone cooks it, and someone plates it before it reaches your table. If any one of them quietly does nothing and just passes an empty plate along, you still get *something* — just not what you asked for.\n\nThat's basically what happened.\n\nUsers started getting weird, inconsistent responses. For the first 30 minutes, I was not worried enough to inspect — the monitoring showed **200 OK for 92% of requests**. In most systems, a 200 status means \"this worked.\" So I assumed everything was fine.\n\nIt wasn't. When users kept reporting bad answers, we finally looked past the status codes and into the actual response payloads — and found a field I hadn't been watching closely: `confidence = 0`\n\n.\n\nHere's what was actually happening under the hood:\n\n```\nRouter LLM   ──fails/times out──▶  falls back silently\nAgent LLM    ──fails/times out──▶  falls back silently\nJudge LLM    ──fails/times out──▶  falls back silently\n                                          │\n                                          ▼\n                          Response: 200 OK, confidence = 0\n                     (looks healthy in the logs — nothing \"crashed\")\n```\n\nEvery single stage of the pipeline was failing and quietly falling back to a default response. No exception. No 500 error. No alert. Just a technically-successful HTTP response carrying a useless answer.\n\nI had built in fallback handling for the case where *one* LLM call in the chain might fail. What I hadn't considered was the entire chain failing **together** — because the actual root cause wasn't in my code at all. Around the same time, we started seeing `429 Too Many Requests`\n\nerrors too, and eventually traced it back to a provider-side incident: Anthropic's status page confirmed the model API itself was degraded that day.\n\nThe fix was to add a **Circuit Breaker** at the controller level — the layer that sits in front of all three LLM calls. Before trusting the pipeline with real traffic, the breaker sends a tiny, cheap probe request (just a single token) to confirm the LLM is actually accepting and responding to calls. If the probe fails, the breaker opens immediately and the whole pipeline fails fast with a clear error, instead of quietly returning broken answers dressed up as successes.\n\nThat's the real danger with multi-agent systems: a failure doesn't always look like a crash. Sometimes it looks like three fallbacks politely agreeing to lie to you.\n\nGenAI is the new fire. Everyone's building on top of an LLM provider — OpenAI, Anthropic, Gemini, a self-hosted model, whatever. And every one of those calls is... a network call to a remote dependency that can be **slow, rate-limited, or down**.\n\nSound familiar?\n\nAn LLM call is basically the *ultimate* late friend:\n\n`429 Too Many Requests`\n\nwhen you hit rate limits.Now picture the classic naive setup:\n\n```\nYour AI App → prompt fails or times out → Retry ×3 → Retry ×3 → already rate-limited LLM provider\n```\n\nRetrying a rate-limited model is like ordering more pizza from a kitchen that's already on fire. Every retry pushes you deeper into the rate limit, your latency explodes, and your token bill quietly grows.\n\nA Circuit Breaker in front of the model client fixes this:\n\n```\n       AI Request\n           │\n           ▼\n     Circuit Breaker\n     ╱      │      ╲\n CLOSED   OPEN   HALF-OPEN\n    │       │        │\n    ▼       ▼        ▼\n  LLM    Fallback  Test call\nProvider            to LLM\n```\n\nAnd the fallback is where GenAI apps get to be clever:\n\n```\nModel unavailable\n├── Return a cached / similar answer\n├── Drop to a smaller, cheaper model\n├── Return a \"please try again shortly\" message\n└── Queue the request for later processing\n```\n\nSo the next time your agent loops and calls the model 40 times in 5 seconds, remember: even the fanciest AI needs a circuit breaker before it keeps knocking on the provider's door.\n\nSame old pattern, brand new fire.\n\nA fallback doesn't mean `return null`\n\n. Please don't.\n\n```\nDependency Down\n├── Return cached data\n├── Queue the request\n├── Drop to a cheaper path\n└── Return a graceful error\n```\n\nThe important rule: **Don't fake success.** If the pizza order failed, don't tell the guest \"Order Confirmed ✅\" just because your fallback needs to return *something*.\n\nA graceful failure is much better than a successful lie.\n\nShould **every exception** open the circuit? No.\n\nA `400 Bad Request`\n\n(the guest sent garbage input) doesn't mean the service is unhealthy. But these are strong signals that it is:\n\n```\nConnection refused\nTimeout\n503 Service Unavailable\n429 Too Many Requests\n```\n\nSo when configuring a breaker, one of the most important questions is:\n\nWhich failures should actually count?\n\nThat decision is application-specific.\n\nThat's the Circuit Breaker.\n\nWhenever you hear **Circuit Breaker**, remember the three states:\n\n```\nCLOSED    → requests flow normally\nOPEN      → requests stop immediately\nHALF-OPEN → a few requests test recovery\n```\n\nOr, in Bangalore house-party terms:\n\n```\nFriend is coming → Everyone waits → Friend keeps failing → STOP WAITING → Party Continues 😎\n```\n\nA Circuit Breaker doesn't fix the failing service. It protects the rest of your system **while that service is failing** — whether that's a pizza shop, a payment API, or your favourite LLM.\n\nA failure is inevitable. A cascading failure doesn't have to be.\n\nFirstly, thanks a lot if you're still reading this far. 🎉\n\nRemember, distributed systems are a bit like that one late friend — the failures are inevitable, but how you *handle* them is entirely up to you. A circuit breaker won't fix a broken service, but it'll stop one bad dependency from taking your whole system (or your whole party) down with it.\n\nThe purpose of this blog was to demystify a pattern that sounds intimidating but is genuinely simple once you see it as three states and a timer. Whether it's a pizza order, a payment API, or an LLM call — the same rule applies: stop waiting, fail fast, and test recovery before trusting again.\n\nThe more you work with distributed systems, the more you realise the most useful ideas are simple:\n\nAnd if the answer is no... **open the circuit**.\n\nI plan to write more on distributed systems patterns and backend engineering — the kind of stuff that shows up in real production systems, not just textbooks.\n\nTill then, go build something that doesn't wait around for late friends. 😄\n\nIf you run an organisation and want me to write or create video tutorials, please do connect with me 🤝", "url": "https://wpnews.pro/news/why-your-llm-calls-need-a-circuit-breaker-a-lesson-from-last-party", "canonical_source": "https://dev.to/aniket762/why-your-llm-calls-need-a-circuit-breaker-a-lesson-from-last-party-3pa4", "published_at": "2026-07-30 21:02:53+00:00", "updated_at": "2026-07-30 21:32:48.864770+00:00", "lang": "en", "topics": ["artificial-intelligence", "large-language-models", "ai-agents", "developer-tools"], "entities": [], "alternates": {"html": "https://wpnews.pro/news/why-your-llm-calls-need-a-circuit-breaker-a-lesson-from-last-party", "markdown": "https://wpnews.pro/news/why-your-llm-calls-need-a-circuit-breaker-a-lesson-from-last-party.md", "text": "https://wpnews.pro/news/why-your-llm-calls-need-a-circuit-breaker-a-lesson-from-last-party.txt", "jsonld": "https://wpnews.pro/news/why-your-llm-calls-need-a-circuit-breaker-a-lesson-from-last-party.jsonld"}}