Circuit Breakers in Production: Timeouts, Retry Storms, and Failure Recovery

A developer has published a per-process Python circuit breaker implementation for pipelines that call multiple LLM providers (OpenAI, Anthropic, Google), addressing the failure modes that arise when timeouts and retries are layered on top of a slow dependency. The design adds a probe lease to the half-open state so a single probe request tests recovery and a stalled prober cannot leave the circuit stuck, with a thread lock protecting state within one process only.

A dependency doesn't have to go completely offline to cause trouble. Sometimes it just gets slow enough that requests pile up, workers stay busy, and retries add even more traffic to a service already struggling. A circuit breaker gives that dependency breathing room—but only if the timeouts and retries around it are configured with the same care. Working with multiple LLM providers OpenAI, Anthropic, Google presents several challenges. Each provider has different failure modes, rate limits, and latency characteristics. Without proper handling, a slow or failing provider can bring down an entire pipeline. The naive approach is to add timeouts and retries. But this creates new problems: The circuit breaker pattern, popularized by Michael Nygard in "Release It ", works like an electrical circuit breaker. When failures exceed a threshold, the circuit "opens" and immediately fails requests without attempting them. Three states: The half-open state is where most implementations go wrong. If the dependency is just coming back, the worst thing you can do is send it the full load again. One probe is enough to learn whether it recovered. Here's a simplified circuit breaker. It is per-process : the lock protects threads in one process, and nothing here coordinates between processes more on that below . python import time from datetime import timedelta from enum import Enum from threading import Lock class CircuitState Enum : CLOSED = "closed" OPEN = "open" HALF OPEN = "half open" class CircuitBreaker: def init self, failure threshold: int = 5, recovery timeout: timedelta = timedelta seconds=30 , probe lease: timedelta = timedelta seconds=10 , : self.failure threshold = failure threshold self.recovery timeout = recovery timeout self.probe lease = probe lease self.state = CircuitState.CLOSED self.failure count = 0 self.opened at = 0.0 self.probe deadline = 0.0 when the current probe's claim expires self.lock = Lock def can execute self - bool: with self.lock: now = time.monotonic if self.state == CircuitState.CLOSED: return True if self.state == CircuitState.OPEN: if now - self.opened at < self.recovery timeout.total seconds : return False Timeout over: this caller claims the probe. self.state = CircuitState.HALF OPEN self.probe deadline = now + self.probe lease.total seconds return True Half-open: one probe is already out. Everyone else fails fast, unless the probe's lease expired the prober never reported back . if now = self.probe deadline: self.probe deadline = now + self.probe lease.total seconds return True return False def record success self : with self.lock: if self.state == CircuitState.HALF OPEN: self.state = CircuitState.CLOSED self.failure count = 0 def record failure self : with self.lock: if self.state == CircuitState.OPEN: return a late result from a call that started before we opened if self.state == CircuitState.HALF OPEN: self. open the probe failed: back to open, restart the timer return self.failure count += 1 if self.failure count = self.failure threshold: self. open def open self : self.state = CircuitState.OPEN self.opened at = time.monotonic Two things changed from the naive version, and both matter: record success or record failure , the circuit would otherwise sit in half-open forever. The lease lets another caller try after a bounded time. The lock above is a thread lock. If you run twenty workers, you have twenty independent breakers, and each one sends its own probe: twenty probes, not one. For many services that is acceptable, and per-process breakers are common because they are simple and add no dependency. But say it plainly: this example gives you one probe per process. If you need one probe for the whole fleet, the claim must be atomic in shared state. A short-lived lease in Redis is one way: Returns True for exactly one caller until the lease expires. claimed = redis.set f"breaker:{service}:probe", worker id, nx=True, ex=10 SET ... NX EX succeeds for only one caller, and the expiry is the lease: if the prober dies, the key vanishes and someone else can claim it. The costs are real, though. Every call now depends on Redis being reachable, so decide what the breaker does when Redis is down usually: fall back to the local breaker . A database row with UPDATE ... WHERE probe claimed until < now works the same way. A breaker can only count failures that the caller actually gives up on. If a call hangs for two minutes, it is neither a success nor a failure for that whole time, and the threshold is never reached while workers pile up behind it. So: Retries are what turn a partial outage into a retry storm. Three habits keep them safe: Retrying also requires the operation to be safe to repeat. For a write, send an idempotency key so a retry after an ambiguous timeout doesn't do the work twice. Failure threshold : The example counts consecutive failures, and a success resets the count. A rolling window of recent calls is better when traffic is mixed, because it can tell "5 failures in a row" from "5 failures out of 10,000". Too low a threshold and you'll open on transient errors. Too high and you'll let too many requests fail first. Recovery timeout : 30 seconds is a reasonable starting point for many APIs, not a rule. For rate-limited services, we use longer timeouts based on the rate limit reset time. What counts as failure : Not all errors should trip the breaker. In the example, we count: Client errors 4xx are the caller's problem, not a service failure. A rate-limit response 429 is a signal to slow down, so handle it with backoff and the provider's retry-after hint instead of counting it toward opening the circuit. An open circuit isn't useful if you just return an error. We use several fallback approaches: Circuit state changes are important signals. We track: Start conservative : It's easier to relax thresholds than to explain cascading failures. Test the open state : Your fallback logic needs as much testing as the happy path. Test half-open with concurrent callers too, and check that only one request goes out. Different breakers for different operations : Read and write operations to the same service might need different thresholds. Document the behavior : Teams need to know what happens when a circuit opens. Circuit breakers are one piece of a resilience strategy. Combined with timeouts, bulkheads, and proper retry policies, they help systems degrade gracefully instead of failing all at once. The primary material behind the pattern, and what to read once the basics land. If the three states now make sense, the next thing to understand is timeouts, because a breaker can only count failures that the caller actually gives up on. After that, bulkheads, which stop one failing dependency from consuming the thread pool everything else needs. Then load shedding, which is what you do when the problem is your own service rather than a dependency. Where should breaker state live? Per-process is the simplest and is often enough: no extra dependency, and each worker protects itself. The cost is that every worker discovers the outage on its own, and sends its own half-open probe. Shared state in Redis or a database gives you one view and one probe for the fleet, but adds a dependency you must handle when it fails. If you share state, make the probe claim atomic. What failure rate should open the breaker? Start from what a healthy dependency looks like, not from a round number. If it normally errors on 1% of calls, a 50% threshold over a meaningful window is a real signal, while 5% is noise that can trip during a routine blip. Should a breaker cover all errors? No. Count the ones that mean the dependency is unhealthy: timeouts, connection failures, 5xx. A 400 means your request was wrong and will fail identically on every retry, and a 429 is a rate limit that backoff handles better. Should I retry inside the breaker? Prefer retrying outside it, with a capped number of attempts, jittered backoff and a retry budget, and stop retrying as soon as the circuit is open. Originally published at www.internalswithzee.com/blog/circuit-breakers https://www.internalswithzee.com/blog/circuit-breakers .