Circuit Breaker Pattern in Go: Stop Cascading Failures

A circuit breaker pattern in Go stops cascading failures by failing fast when a dependency crosses a failure threshold, protecting the system from slow or unhealthy services. The pattern uses three states—closed, open, and half-open—and is distinct from timeouts and retries, which solve different problems. Proper placement and configuration are critical to avoid introducing new failure modes.

Circuit Breaker Pattern in Go: Stop Cascading Failures Stop cascading failures in Go microservices. A circuit breaker stops your Go service from hammering a failing dependency, preventing cascading failures that consume goroutines, sockets, and memory until the entire system collapses. The hard part is not the state machine. It is deciding where the breaker belongs, what counts as failure, how it interacts with timeouts and retries, and what your service should do when the circuit is open. In Go, the circuit breaker pattern is especially useful around outbound calls: HTTP APIs, payment gateways, search services, email providers, LLM gateways, internal microservices, and other dependencies that can become slow, overloaded, or partially unavailable. Used well, a circuit breaker reduces cascading failures. Used badly, it becomes another obscure failure mode. What Problem Does a Circuit Breaker Solve? Distributed systems rarely fail cleanly. A dependency might not be fully down. It might be: - returning 500 errors - returning 429 rate limit responses - accepting TCP connections but never replying - responding in 30 seconds instead of 300 milliseconds - failing only for some requests - overloaded because every client is retrying at once The worst case is often not a hard failure. It is a slow dependency. Slow calls consume goroutines, sockets, database connections, memory, and worker capacity. If your service keeps waiting on a dependency that is already unhealthy, your service can become unhealthy too. A circuit breaker prevents that by failing fast after the dependency crosses a failure threshold. Instead of doing this forever: php request - call dependency - wait - timeout - retry - wait - fail the service eventually does this: php request - circuit open - return fallback or error immediately That fast failure is not always pleasant, but it is predictable. Predictable failure is easier to operate than a slow collapse. The Three Circuit Breaker States Most circuit breakers use three states. Closed The circuit is closed during normal operation. Requests are allowed through. The breaker records successes and failures. If the number or ratio of failures crosses a threshold, the breaker opens. Closed does not mean “safe forever.” It means “traffic is currently allowed.” Open The circuit is open when the dependency is considered unhealthy. Requests are rejected immediately. The service should return a fallback, cached response, degraded response, or a clear upstream error. Open does not fix the dependency. It gives the dependency time to recover and protects the caller from wasting resources. Half-Open After a cool-down period, the breaker enters a half-open state. Only a limited number of trial requests are allowed through. If they succeed, the breaker closes. If they fail, the breaker opens again. Half-open is important because it avoids two bad extremes: - never trying the dependency again - sending full traffic back too quickly The state transitions look like this: Circuit Breaker vs Timeout vs Retry A common mistake is treating circuit breakers, retries, and timeouts as interchangeable. They are related, but they solve different problems. Timeout A timeout limits how long one operation can run. In Go, this usually means passing a context.Context with a deadline or timeout into the outbound call. A timeout answers this question: How long am I willing to wait for this one call? Retry A retry repeats an operation when the failure might be temporary. Retries are useful for short network glitches, temporary 503 responses, connection resets, and other transient failures. A retry answers this question: Should I try this call again? Circuit Breaker A circuit breaker stops calls when the dependency is probably unhealthy. It answers this question: Should I call this dependency at all right now? Rate Limiter A rate limiter controls how much traffic is allowed over time. It answers this question: How much traffic should this caller send? Bulkhead A bulkhead isolates resources so one dependency cannot consume everything. It answers this question: How much of my service can this dependency damage? These patterns are strongest when used together. A circuit breaker without timeouts is weak. Retries without jitter can create retry storms. A fallback without metrics can hide an outage. When to Use a Circuit Breaker in Go Use a circuit breaker when your service calls a dependency that can fail independently from your service. Good candidates include: - external HTTP APIs - payment processors - email and SMS providers - search services - recommendation services - LLM inference gateways - internal microservice endpoints - third-party SaaS APIs - slow or overloaded read-side services Circuit breakers are especially useful when the caller can degrade gracefully. For example: - return cached product data - skip a recommendation block - mark a payment provider as temporarily unavailable - queue work for later - return a partial response - fail fast with a clear temporary error The important question is not “can this call fail?” Everything can fail. The better question is: If this dependency is failing, should we continue sending full traffic to it? If the answer is no, a circuit breaker may help. When Not to Use a Circuit Breaker Do not add a circuit breaker to every function just because the pattern sounds responsible. A circuit breaker is usually not useful for: - local in-process function calls - simple CRUD inside a monolith - validation logic - deterministic business rules - CPU-only local operations - code paths where no useful fallback exists - write operations that are not idempotent - dependencies already protected by a stronger workflow layer A circuit breaker also does not replace basic hygiene: - set timeouts - propagate context - use connection pools correctly - handle errors explicitly - make retries safe - observe failure rates A bad circuit breaker can make a system harder to reason about. It can hide the real problem, reject traffic too aggressively, or create confusing behavior during recovery. The slightly opinionated rule is simple: Add circuit breakers at dependency boundaries, not everywhere. Choosing a Go Circuit Breaker Library You can implement a basic circuit breaker yourself, but most production Go services should use a library. The most common simple choice is sony/gobreaker . It gives you: - closed, open, and half-open states - configurable failure thresholds - configurable open-state timeout - state change callbacks - request counters - generic support in v2 - a small API surface For larger resilience pipelines, you may also look at libraries that compose multiple policies, such as retry, timeout, fallback, rate limiting, bulkhead isolation, and circuit breaking. That can be useful when you want a single resilience layer around an operation. For many Go services, though, gobreaker is enough. Go Circuit Breaker Packages Compared Go does not include a built-in circuit breaker in the standard library. In practice, you usually choose between a small circuit breaker library, a larger resilience framework, or an older Hystrix-style package. For most new Go services, the decision is simple: - use sony/gobreaker if you want a small, focused circuit breaker - use failsafe-go if you want circuit breakers composed with retries, timeouts, fallbacks, bulkheads, rate limits, and other resilience policies - avoid starting new projects on hystrix-go unless you already have legacy code using it | Package | Best for | Strengths | Tradeoffs | |---|---|---|---| sony/gobreaker/v2 | Simple circuit breakers around HTTP/RPC clients | Small API, generic v2 support, clear state model, easy to wrap dependency clients | Only solves circuit breaking; retries, timeouts, and fallbacks must be composed separately | failsafe-go | Full resilience policy composition | Retry, fallback, circuit breaker, timeout, bulkhead, rate limiter, cache, hedge, adaptive limiter, and adaptive throttler policies | More concepts to learn; heavier than needed if you only want a basic breaker | afex/hystrix-go | Legacy Hystrix-style systems | Familiar Hystrix concepts, command-style execution, historical usage | Older design; not the best default for new Go services | go-kit/kit/circuitbreaker | Go kit endpoint-based services | Fits Go kit middleware style and endpoint architecture | Mostly useful if your service already uses Go kit | cep21/circuit | Hystrix-like circuit breaker behavior | More featureful Hystrix-style approach | Less common as the simple default; may be more than needed for small services | My default recommendation is boring on purpose: start with sony/gobreaker/v2 when you only need a circuit breaker. Reach for failsafe-go when you want to express a complete resilience policy in one place. That split keeps the architecture clean. A small service client does not need a full resilience framework just to stop calling a failing dependency. But a gateway, aggregator, API client SDK, or high-traffic integration layer may benefit from composed policies. Installing gobreaker Use the v2 package for new code: go get github.com/sony/gobreaker/v2 Then import it: import "github.com/sony/gobreaker/v2" A Basic Circuit Breaker in Go Here is a small example around an HTTP call. package main import "context" "errors" "fmt" "io" "net/http" "time" "github.com/sony/gobreaker/v2" var ErrTemporaryUnavailable = errors.New "dependency temporarily unavailable" type UserClient struct { baseURL string http http.Client cb gobreaker.CircuitBreaker byte } func NewUserClient baseURL string UserClient { settings := gobreaker.Settings{ Name: "user-service", MaxRequests: 3, Interval: 30 time.Second, Timeout: 10 time.Second, ReadyToTrip: func counts gobreaker.Counts bool { return counts.ConsecutiveFailures = 5 }, OnStateChange: func name string, from gobreaker.State, to gobreaker.State { fmt.Printf "circuit breaker %s changed from %s to %s\n", name, from, to }, } return &UserClient{ baseURL: baseURL, http: &http.Client{ Timeout: 3 time.Second, }, cb: gobreaker.NewCircuitBreaker byte settings , } } func c UserClient GetUser ctx context.Context, userID string byte, error { result, err := c.cb.Execute func byte, error { req, err := http.NewRequestWithContext ctx, http.MethodGet, c.baseURL+"/users/"+userID, nil, if err = nil { return nil, err } resp, err := c.http.Do req if err = nil { return nil, err } defer resp.Body.Close if resp.StatusCode = 500 { return nil, fmt.Errorf "user service returned %d", resp.StatusCode } if resp.StatusCode == http.StatusNotFound { return nil, fmt.Errorf "user not found" } if resp.StatusCode = 400 { return nil, fmt.Errorf "user service client error: %d", resp.StatusCode } return io.ReadAll resp.Body } if errors.Is err, gobreaker.ErrOpenState { return nil, ErrTemporaryUnavailable } if errors.Is err, gobreaker.ErrTooManyRequests { return nil, ErrTemporaryUnavailable } return result, err } This is not a complete production client, but it shows the shape: - the breaker wraps the outbound call - the HTTP request receives a context - the HTTP client has a timeout - server-side failures count as breaker failures - open circuit errors are mapped into an application error Configuring gobreaker Settings The key settings are worth understanding. Name Name identifies the breaker. Use a stable, specific name: payment-api search-service llm-gateway user-service Avoid vague names like: http-client external-call default You will want this name in logs and metrics. MaxRequests MaxRequests controls how many requests are allowed while the breaker is half-open. A small number is usually safer. The purpose of half-open is to test recovery, not to send full traffic immediately. Interval Interval controls when internal counts are cleared while the breaker is closed. If it is zero, counts are not cleared automatically. A non-zero interval gives the breaker a rolling-ish memory window, although it is not the same as a full sliding window implementation. Timeout Timeout controls how long the breaker stays open before moving to half-open. If the timeout is too short, your service will keep probing a dependency that has not recovered. If it is too long, recovery will be delayed. Start with something conservative, such as 10 to 30 seconds, then tune from production metrics. ReadyToTrip ReadyToTrip decides when the breaker should open. A simple rule is consecutive failures: ReadyToTrip: func counts gobreaker.Counts bool { return counts.ConsecutiveFailures = 5 } That is easy to reason about, but it may not be right for high-volume services. Another option is failure ratio after a minimum number of requests: ReadyToTrip: func counts gobreaker.Counts bool { total := counts.Requests failures := counts.TotalFailures if total < 20 { return false } return float64 failures /float64 total = 0.5 } This avoids opening the circuit after a tiny sample size. OnStateChange OnStateChange is where you should emit logs or metrics. At minimum, record: - breaker name - old state - new state - timestamp For production systems, expose breaker state as a metric. Logs are useful for debugging, but metrics are better for alerting and dashboards. IsSuccessful IsSuccessful lets you decide which errors count as failures. This is important. Not every error should open the breaker. For example, a 404 Not Found from a user service may be a valid business result. A 400 Bad Request might be the caller’s fault, not the dependency’s fault. A 503 Service Unavailable , timeout, connection reset, or 429 Too Many Requests may be a real dependency health signal. Be careful here. Counting the wrong errors is one of the easiest ways to build a noisy circuit breaker. What Should Count as Failure? This is where engineering judgement matters. Usually count these as failures: - network timeouts - connection refused - connection reset - HTTP 500 - HTTP 502 - HTTP 503 - HTTP 504 - repeated 429 responses - malformed responses from the dependency - context deadline exceeded during the outbound call Usually do not count these as dependency failures: - validation errors - local serialization errors - expected 404 responses - caller-side authorization failures - business rule rejections - user input errors The breaker should represent dependency health, not general application failure. Circuit Breakers and context.Context In Go, circuit breakers should not replace context.Context . A circuit breaker decides whether to attempt a call. A context controls how long that call may run and whether it should stop when the caller is gone. A good outbound call should usually have both: ctx, cancel := context.WithTimeout parentCtx, 2 time.Second defer cancel data, err := client.GetUser ctx, userID The context should flow through the call chain: php incoming request context - service method - client method - HTTP request - dependency Avoid creating detached background contexts inside request-scoped code. If the user request is canceled, the downstream work should usually stop too. The calm rule is: The breaker protects the system. The context protects the request. You normally need both. Circuit Breakers and Retries Retries and circuit breakers can work well together, but the order matters. The safest default is: timeout per attempt retry with backoff and jitter circuit breaker around the dependency call But there is no universal answer. Think about what you want to count. If each retry attempt passes through the breaker, one user request can contribute multiple failures. That may open the breaker faster, which can be good or bad. If the breaker wraps the whole retry operation, the breaker sees one final success or failure per user request. That is calmer, but it may hide the number of failed attempts. For many application services, this shape is reasonable: php user request - circuit breaker - retry policy - one HTTP attempt with timeout That means the breaker tracks whether the dependency operation ultimately worked for the caller. For lower-level clients, this shape can also make sense: php user request - retry policy - circuit breaker - one HTTP attempt with timeout That means the breaker protects each attempt. The more important rule is this: Do not retry blindly. Use: - a small maximum retry count - exponential backoff - jitter - per-attempt timeouts - an overall request deadline - idempotency for writes - metrics for retry attempts Without those, retries can turn a small outage into a larger one. For a deeper treatment of retry safety, see Idempotency in Distributed Systems That Actually Works https://www.glukhov.org/app-architecture/integration-patterns/idempotency-in-distributed-systems/ . Circuit Breakers and Idempotency Circuit breakers often appear next to retries, and retries raise the question of idempotency. For read operations, retrying is usually safe. For write operations, retrying can be dangerous. Consider this payment call: POST /charge If the request times out, did the payment fail? Maybe. Did it succeed but the response was lost? Also maybe. If you retry without an idempotency key, you might charge twice. For write operations, use one or more of these: - idempotency keys - request IDs - operation IDs - unique constraints - transactional outbox - workflow orchestration - explicit reconciliation A circuit breaker can stop you from continuing to call a failing payment provider, but it cannot make unsafe retries safe. Circuit Breakers and Fallbacks When the circuit is open, your service needs a plan. Possible fallback strategies include: - return cached data - return stale data with a warning - omit a non-critical section - queue work for later - switch to another provider - return a temporary error - show degraded functionality - fail the request quickly A fallback should be honest. For example, this is usually good: { "status": "temporary unavailable", "message": "Recommendations are temporarily unavailable" } This is risky: { "recommendations": } An empty list may look like a valid result. It can hide an outage, confuse users, and make debugging harder. Silent fallbacks are tempting. They are also dangerous. Circuit Breakers and Observability A circuit breaker without observability is mostly a surprise generator. Track at least these metrics: - current breaker state - state changes - calls allowed - calls rejected - successes - failures - timeouts - fallback responses - retry attempts - downstream latency - downstream status codes Useful labels include: - breaker name - dependency name - operation name - status class - error category Avoid high-cardinality labels such as user ID, full URL, request ID, or raw error messages. You should be able to answer these questions from dashboards: - Which circuit breakers are open right now? - How often do they open? - Which dependency caused the opening? - Are users seeing fallback responses? - Did latency improve after the breaker opened? - Did retry volume spike before the breaker opened? - Did the dependency recover? If you cannot observe the breaker, you cannot tune it. For structured logging that pairs well with metrics, see Structured Logging in Go with slog https://www.glukhov.org/observability/logging/structured-logging-go-slog/ . A More Production-Friendly HTTP Client Shape For real services, avoid scattering circuit breaker logic across handlers. Create a small client package around the dependency. Example structure: internal/ userservice/ client.go errors.go metrics.go The handler should not know the details of gobreaker. It should depend on a domain-level client method: type UserService interface { GetUser ctx context.Context, userID string User, error } Then the implementation can contain: - HTTP request creation - context propagation - breaker execution - status code handling - response decoding - metrics - error mapping This keeps the resilience policy close to the dependency boundary. For more on error classification at boundaries, see Go Error Handling Architecture: Boundaries and Patterns https://www.glukhov.org/app-architecture/code-architecture/go-error-handling-architecture/ . Where Circuit Breakers Fit in Application Architecture The circuit breaker pattern belongs at integration boundaries. In a Go application, that usually means: Keep the breaker out of business logic when possible. The business layer should understand domain errors like: payment provider unavailable recommendations unavailable profile service timeout It should not need to understand gobreaker states. This separation keeps the architecture clean: - transport concerns stay in clients - resilience policy stays near dependencies - domain logic stays readable - handlers stay thin - tests are easier to write Common Mistakes Mistake 1: No Timeout A circuit breaker does not magically stop slow calls unless the calls return. If the outbound operation can hang forever, the breaker may not see a failure quickly enough. Always use timeouts. Mistake 2: One Global Breaker for Everything Do not use one breaker for all dependencies. A failing email provider should not open the circuit for your payment provider. A slow search endpoint should not block user profile calls. Use separate breakers for separate dependency operations when their failure modes differ. Mistake 3: Counting Caller Errors as Dependency Failures If your service sends bad input and receives 400 Bad Request , that is usually not a downstream outage. Do not train the breaker on your own bugs. Mistake 4: Retrying Non-Idempotent Writes Retries are not free. They can duplicate writes, payments, messages, or side effects. Make writes idempotent before retrying them. Mistake 5: Hiding Outages Behind Fallbacks Fallbacks should degrade gracefully, not falsify reality. If a dependency is down, your metrics and logs should make that obvious. Mistake 6: Tuning Without Production Data Thresholds copied from examples are only starting points. Tune based on: - request volume - normal error rate - dependency latency - user impact - recovery time - fallback quality Mistake 7: Using Circuit Breakers Instead of Capacity Management A circuit breaker is not a substitute for: - load shedding - rate limiting - queue limits - autoscaling - database tuning - connection pool limits - upstream quotas It is one part of a resilience strategy. Practical Defaults For a typical Go service calling an internal HTTP dependency, a reasonable starting point might be: HTTP client timeout: 2 to 5 seconds per-request context timeout: based on caller SLA breaker failure rule: 5 consecutive failures or 50 percent failure after 20 requests open timeout: 10 to 30 seconds half-open requests: 1 to 5 retry count: 1 to 3 attempts retry backoff: exponential with jitter These are not universal values. They are safe-ish starting points. For user-facing APIs, keep total latency budgets tight. For background jobs, you may tolerate longer waits. For payment providers, be much more careful with retries and idempotency. Circuit Breaker Checklist Before adding a circuit breaker, answer these questions: - What dependency is being protected? - What operation is being protected? - What errors count as dependency failure? - What errors should be ignored by the breaker? - What timeout applies to each call? - Are retries allowed? - Are writes idempotent? - What happens when the circuit is open? - Is there a fallback? - Is the fallback visible in metrics? - Who gets alerted if the circuit keeps opening? - How will the breaker be tuned after deployment? If you cannot answer these, adding a breaker may create more confusion than resilience. Testing Circuit Breakers in Go Test behavior, not the internal state machine of the library. Useful tests include: - dependency succeeds and response is returned - dependency fails repeatedly and circuit opens - open circuit returns a temporary error - client-side validation errors do not trip the breaker - context timeout is respected - fallback response is returned when expected - metrics are emitted on state changes Use fake HTTP servers for integration-style tests: server := httptest.NewServer http.HandlerFunc func w http.ResponseWriter, r http.Request { http.Error w, "unavailable", http.StatusServiceUnavailable } defer server.Close For unit tests, hide the dependency behind an interface and inject a fake implementation. Keep tests deterministic. Avoid sleeping for long real durations. Configure short breaker timeouts in tests. For more on testing concurrent Go code with fake time and isolated bubbles, see Testing Concurrent Go Code with testing/synctest https://www.glukhov.org/app-architecture/testing-architecture/testing-concurrent-go-code-synctest/ . Should You Build Your Own Circuit Breaker? Building a small circuit breaker is a good learning exercise. It helps you understand the state machine. For production code, prefer a maintained library unless your needs are very specific. A production breaker needs to handle: - concurrency - state transitions - counters - half-open probes - callbacks - custom failure classification - race-free behavior - predictable error handling That is not impossible, but it is easy to get subtly wrong. The boring library is usually the better choice. Conclusion The circuit breaker pattern is not magic reliability dust. In Go, it works best when it is part of a small, explicit resilience stack: context timeout + retry with backoff and jitter + circuit breaker + fallback + metrics The pattern is most useful at dependency boundaries, especially around remote services that can become slow or partially unavailable. Use it to stop cascading failures. Use it to fail fast when a dependency is clearly unhealthy. Use it to give overloaded systems room to recover. But do not use it as an excuse to ignore timeouts, idempotency, observability, or clean architecture. A good circuit breaker makes failure clearer and cheaper. A bad one just makes failure more mysterious. References Go Microservices for AI/ML Orchestration https://www.glukhov.org/app-architecture/integration-patterns/go-microservices-for-ai-ml-orchestration-patterns/ — broader orchestration context where circuit breakers fit Saga Pattern in Distributed Transactions https://www.glukhov.org/app-architecture/integration-patterns/saga-transactions-in-microservices/ — distributed transaction patterns that pair with circuit breakers Idempotency in Distributed Systems https://www.glukhov.org/app-architecture/integration-patterns/idempotency-in-distributed-systems/ — retry safety and idempotent operations Transactional Outbox Pattern in Go https://www.glukhov.org/app-architecture/integration-patterns/transactional-outbox-pattern-go/ — reliable event delivery alongside resilience patterns Go Error Handling Architecture https://www.glukhov.org/app-architecture/code-architecture/go-error-handling-architecture/ — error classification at dependency boundaries Testing Concurrent Go Code with synctest https://www.glukhov.org/app-architecture/testing-architecture/testing-concurrent-go-code-synctest/ — testing async behavior with circuit breakers Structured Logging in Go with slog https://www.glukhov.org/observability/logging/structured-logging-go-slog/ — observability alongside circuit breakers github.com/sony/gobreaker/v2 — official gobreaker v2 package Go Context Cancellation and Timeouts https://www.glukhov.org/app-architecture/code-architecture/go-context-cancellation-timeouts/ — context patterns that pair with circuit breakers