{"slug": "circuit-breakers-in-production-timeouts-retry-storms-and-failure-recovery", "title": "Circuit Breakers in Production: Timeouts, Retry Storms, and Failure Recovery", "summary": "A developer has published a per-process Python circuit breaker implementation for pipelines that call multiple LLM providers (OpenAI, Anthropic, Google), addressing the failure modes that arise when timeouts and retries are layered on top of a slow dependency. The design adds a probe lease to the half-open state so a single probe request tests recovery and a stalled prober cannot leave the circuit stuck, with a thread lock protecting state within one process only.", "body_md": "A dependency doesn't have to go completely offline to cause trouble. Sometimes it just gets slow enough that requests pile up, workers stay busy, and retries add even more traffic to a service already struggling. A circuit breaker gives that dependency breathing room—but only if the timeouts and retries around it are configured with the same care.\n\nWorking with multiple LLM providers (OpenAI, Anthropic, Google) presents several challenges. Each provider has different failure modes, rate limits, and latency characteristics. Without proper handling, a slow or failing provider can bring down an entire pipeline.\n\nThe naive approach is to add timeouts and retries. But this creates new problems:\n\nThe circuit breaker pattern, popularized by Michael Nygard in \"Release It!\", works like an electrical circuit breaker. When failures exceed a threshold, the circuit \"opens\" and immediately fails requests without attempting them.\n\nThree states:\n\nThe half-open state is where most implementations go wrong. If the dependency is just coming back, the worst thing you can do is send it the full load again. One probe is enough to learn whether it recovered.\n\nHere's a simplified circuit breaker. It is **per-process**: the lock protects threads in one process, and nothing here coordinates between processes (more on that below).\n\n``` python\nimport time\nfrom datetime import timedelta\nfrom enum import Enum\nfrom threading import Lock\n\nclass CircuitState(Enum):\n    CLOSED = \"closed\"\n    OPEN = \"open\"\n    HALF_OPEN = \"half_open\"\n\nclass CircuitBreaker:\n    def __init__(\n        self,\n        failure_threshold: int = 5,\n        recovery_timeout: timedelta = timedelta(seconds=30),\n        probe_lease: timedelta = timedelta(seconds=10),\n    ):\n        self.failure_threshold = failure_threshold\n        self.recovery_timeout = recovery_timeout\n        self.probe_lease = probe_lease\n\n        self.state = CircuitState.CLOSED\n        self.failure_count = 0\n        self.opened_at = 0.0\n        self.probe_deadline = 0.0   # when the current probe's claim expires\n        self.lock = Lock()\n\n    def can_execute(self) -> bool:\n        with self.lock:\n            now = time.monotonic()\n\n            if self.state == CircuitState.CLOSED:\n                return True\n\n            if self.state == CircuitState.OPEN:\n                if now - self.opened_at < self.recovery_timeout.total_seconds():\n                    return False\n                # Timeout over: this caller claims the probe.\n                self.state = CircuitState.HALF_OPEN\n                self.probe_deadline = now + self.probe_lease.total_seconds()\n                return True\n\n            # Half-open: one probe is already out. Everyone else fails fast,\n            # unless the probe's lease expired (the prober never reported back).\n            if now >= self.probe_deadline:\n                self.probe_deadline = now + self.probe_lease.total_seconds()\n                return True\n            return False\n\n    def record_success(self):\n        with self.lock:\n            if self.state == CircuitState.HALF_OPEN:\n                self.state = CircuitState.CLOSED\n            self.failure_count = 0\n\n    def record_failure(self):\n        with self.lock:\n            if self.state == CircuitState.OPEN:\n                return   # a late result from a call that started before we opened\n\n            if self.state == CircuitState.HALF_OPEN:\n                self._open()   # the probe failed: back to open, restart the timer\n                return\n\n            self.failure_count += 1\n            if self.failure_count >= self.failure_threshold:\n                self._open()\n\n    def _open(self):\n        self.state = CircuitState.OPEN\n        self.opened_at = time.monotonic()\n```\n\nTwo things changed from the naive version, and both matter:\n\n`record_success` or `record_failure`, the circuit would otherwise sit in half-open forever. The lease lets another caller try after a bounded time.\nThe lock above is a thread lock. If you run twenty workers, you have twenty independent breakers, and each one sends its own probe: twenty probes, not one. For many services that is acceptable, and per-process breakers are common because they are simple and add no dependency. But say it plainly: this example gives you one probe per process.\n\nIf you need one probe for the whole fleet, the claim must be atomic in shared state. A short-lived lease in Redis is one way:\n\n```\n# Returns True for exactly one caller until the lease expires.\nclaimed = redis.set(f\"breaker:{service}:probe\", worker_id, nx=True, ex=10)\n```\n\n`SET ... NX EX` succeeds for only one caller, and the expiry is the lease: if the prober dies, the key vanishes and someone else can claim it. The costs are real, though. Every call now depends on Redis being reachable, so decide what the breaker does when Redis is down (usually: fall back to the local breaker). A database row with `UPDATE ... WHERE probe_claimed_until < now()` works the same way.\n\nA breaker can only count failures that the caller actually gives up on. If a call hangs for two minutes, it is neither a success nor a failure for that whole time, and the threshold is never reached while workers pile up behind it. So:\n\nRetries are what turn a partial outage into a retry storm. Three habits keep them safe:\n\nRetrying also requires the operation to be safe to repeat. For a write, send an idempotency key so a retry after an ambiguous timeout doesn't do the work twice.\n\n**Failure threshold**: The example counts consecutive failures, and a success resets the count. A rolling window of recent calls is better when traffic is mixed, because it can tell \"5 failures in a row\" from \"5 failures out of 10,000\". Too low a threshold and you'll open on transient errors. Too high and you'll let too many requests fail first.\n\n**Recovery timeout**: 30 seconds is a reasonable starting point for many APIs, not a rule. For rate-limited services, we use longer timeouts based on the rate limit reset time.\n\n**What counts as failure**: Not all errors should trip the breaker. In the example, we count:\n\nClient errors (4xx) are the caller's problem, not a service failure. A rate-limit response (429) is a signal to slow down, so handle it with backoff and the provider's retry-after hint instead of counting it toward opening the circuit.\n\nAn open circuit isn't useful if you just return an error. We use several fallback approaches:\n\nCircuit state changes are important signals. We track:\n\n**Start conservative**: It's easier to relax thresholds than to explain cascading failures.\n\n**Test the open state**: Your fallback logic needs as much testing as the happy path. Test half-open with concurrent callers too, and check that only one request goes out.\n\n**Different breakers for different operations**: Read and write operations to the same service might need different thresholds.\n\n**Document the behavior**: Teams need to know what happens when a circuit opens.\n\nCircuit breakers are one piece of a resilience strategy. Combined with timeouts, bulkheads, and proper retry policies, they help systems degrade gracefully instead of failing all at once.\n\nThe primary material behind the pattern, and what to read once the basics land.\n\nIf the three states now make sense, the next thing to understand is timeouts, because a breaker can only count failures that the caller actually gives up on. After that, bulkheads, which stop one failing dependency from consuming the thread pool everything else needs. Then load shedding, which is what you do when the problem is your own service rather than a dependency.\n\n**Where should breaker state live?**\n\nPer-process is the simplest and is often enough: no extra dependency, and each worker protects itself. The cost is that every worker discovers the outage on its own, and sends its own half-open probe. Shared state in Redis or a database gives you one view and one probe for the fleet, but adds a dependency you must handle when it fails. If you share state, make the probe claim atomic.\n\n**What failure rate should open the breaker?**\n\nStart from what a healthy dependency looks like, not from a round number. If it normally errors on 1% of calls, a 50% threshold over a meaningful window is a real signal, while 5% is noise that can trip during a routine blip.\n\n**Should a breaker cover all errors?**\n\nNo. Count the ones that mean the dependency is unhealthy: timeouts, connection failures, 5xx. A 400 means your request was wrong and will fail identically on every retry, and a 429 is a rate limit that backoff handles better.\n\n**Should I retry inside the breaker?**\n\nPrefer retrying outside it, with a capped number of attempts, jittered backoff and a retry budget, and stop retrying as soon as the circuit is open.\n\n*Originally published at [www.internalswithzee.com/blog/circuit-breakers](https://www.internalswithzee.com/blog/circuit-breakers).*", "url": "https://wpnews.pro/news/circuit-breakers-in-production-timeouts-retry-storms-and-failure-recovery", "canonical_source": "https://dev.to/zeeadil/circuit-breakers-in-production-timeouts-retry-storms-and-failure-recovery-4cj1", "published_at": "2026-11-15 19:40:46+00:00", "updated_at": "2026-11-15 19:51:05.620418+00:00", "lang": "en", "topics": ["ai-infrastructure", "large-language-models", "ai-tools", "mlops"], "entities": ["OpenAI", "Anthropic", "Google", "Michael Nygard"], "also_reported_by": [], "alternates": {"html": "https://wpnews.pro/news/circuit-breakers-in-production-timeouts-retry-storms-and-failure-recovery", "markdown": "https://wpnews.pro/news/circuit-breakers-in-production-timeouts-retry-storms-and-failure-recovery.md", "text": "https://wpnews.pro/news/circuit-breakers-in-production-timeouts-retry-storms-and-failure-recovery.txt", "jsonld": "https://wpnews.pro/news/circuit-breakers-in-production-timeouts-retry-storms-and-failure-recovery.jsonld"}}