Multi-provider LLM resilience in Python without provider-specific code

A developer created llm-api-resilience, a Python library that provides retries, ordered failover, circuit breakers, and checkpoint recovery across multiple LLM providers without provider-specific code. The library builds on llm-api-adapter to normalize differences between OpenAI, Anthropic, and Google APIs, allowing resilience logic to be defined once and applied to any provider.

OpenAI, Anthropic, and Google expose different APIs, message formats, tool-calling conventions, error types, and response structures. That difference is manageable while an application uses only one provider. It becomes more expensive when the application needs retries, circuit breakers, fallback routes, observability, and recovery across several providers. Without a shared abstraction, resilience logic tends to be implemented repeatedly: OpenAI integration ├── request conversion ├── retry logic ├── error classification ├── circuit breaker └── tool-call recovery Anthropic integration ├── request conversion ├── retry logic ├── error classification ├── circuit breaker └── tool-call recovery Google integration ├── request conversion ├── retry logic ├── error classification ├── circuit breaker └── tool-call recovery I did not want to build resilience three times. I wanted to define the recovery policy once and apply it across every provider supported by the application. That became llm-api-resilience , a Python library for retries, ordered failover, circuit breakers, and checkpoint recovery across multiple LLM providers. pip install llm-api-resilience Once the provider adapters are configured, an ordered fallback plan takes only a few lines: python from llm api resilience import RecoveryPlan, ResilientLLM, Route llm = ResilientLLM RecoveryPlan Route "openai-primary", openai adapter , Route "anthropic-backup", anthropic adapter , Route "google-last-resort", google adapter , response = llm.chat {"role": "user", "content": "Explain circuit breakers briefly."} print response.selected route print response.content llm-api-resilience llm-api-adapter The library is built on top of llm-api-adapter https://github.com/Inozem/llm api adapter , which provides one interface for OpenAI, Anthropic, and Google. Because provider-specific differences are handled by the adapter, the resilience layer can operate on a shared contract instead of understanding three separate APIs. The result is a Python library with: Before adding resilience, I had already built llm-api-adapter to normalize the main differences between provider APIs. The application can create adapters for different providers and call the same chat method: python import os from llm api adapter.universal adapter import UniversalLLMAPIAdapter openai adapter = UniversalLLMAPIAdapter organization="openai", model="gpt-5.6-sol", api key=os.environ "OPENAI API KEY" , anthropic adapter = UniversalLLMAPIAdapter organization="anthropic", model="claude-sonnet-5", api key=os.environ "ANTHROPIC API KEY" , google adapter = UniversalLLMAPIAdapter organization="google", model="gemini-3.5-flash", api key=os.environ "GOOGLE API KEY" , messages = { "role": "user", "content": "Reply with exactly: OK", } adapters = { "openai": openai adapter, "anthropic": anthropic adapter, "google": google adapter, } for provider, adapter in adapters.items : response = adapter.chat messages=messages, max tokens=64, print provider, adapter.model, "- ", response.content, The important part is not only that the method names are the same. The adapter creates a stable boundary around: The resilience layer does not need to know how Anthropic represents a tool result, which OpenAI parameter controls structured output, or how Google formats its request. It only needs to understand the adapter contract. Once that contract existed, models from different providers could become routes in the same recovery plan. A recovery plan contains an ordered collection of routes. Each route contains an adapter, a RoutePolicy for retries, timeouts, and backoff, and an optional CircuitBreaker . python from llm api resilience import CircuitBreaker, RecoveryPlan, ResilientLLM, Route, RoutePolicy, primary breaker = CircuitBreaker failure threshold=3, cooldown s=30, llm = ResilientLLM RecoveryPlan Route "openai-primary", openai adapter, policy=RoutePolicy timeout s=15, max attempts=2, backoff s=0.25, , breaker=primary breaker, , Route "anthropic-backup", anthropic adapter, policy=RoutePolicy timeout s=20, max attempts=1, , , Route "google-last-resort", google adapter, policy=RoutePolicy timeout s=30, max attempts=1, , , response = llm.chat messages, max tokens=128, print "Selected route:", response.selected route print "Response:", response.content The application sends one normalized request. The adapter handles provider-specific integration. The resilience layer decides which route to use and what to do when it fails. Application ↓ llm-api-resilience ↓ llm-api-adapter ↓ OpenAI / Anthropic / Google The application does not need separate fallback branches that rebuild messages, tools, and parameters for every provider. Adding a retry loop around every exception is easy. Deciding which failures should actually be retried is harder. A timeout may succeed on the next attempt. A server error may disappear after a short delay. A rate limit may require backoff or a switch to another route. An invalid API key will not be fixed by retrying. Neither will an invalid schema or an incorrectly configured tool. In the current default classifier, only normalized timeout, rate-limit, and server errors are retryable: from llm api adapter.errors import LLMAPIRateLimitError, LLMAPIServerError, LLMAPITimeoutError, from llm api resilience import DefaultFailureClassifier classifier = DefaultFailureClassifier print error type. name for error type in classifier.retryable errors "LLMAPITimeoutError", "LLMAPIRateLimitError", "LLMAPIServerError", Authentication, configuration, invalid tool or schema errors, and ordinary ValueError failures are not retried by default. This is another benefit of placing resilience above a provider adapter. Each provider exposes different exceptions, but the recovery layer does not need three unrelated classification systems. Provider-specific failures are mapped into shared error types before the recovery policy evaluates them. A route can receive multiple attempts before the recovery plan selects the next route. primary route ↓ first attempt fails ↓ retry policy allows another attempt ↓ retry fails ↓ route is exhausted ↓ backup route selected Failover is ordered rather than random. This allows the application to express preferences such as: But every attempt has a cost. Retries add latency and token usage. A backup model may have a different price, context window, or behavioral profile. A resilience policy should therefore not mean “keep trying until something works.” It should define how much latency, cost, and model variation the application is willing to accept. Retries help with isolated errors. They are less useful when a route is already degraded. Without a circuit breaker, every new request may wait for the same route to fail several times before moving to a healthy provider. A circuit breaker remembers recent route health and increments its failure count only for retryable errors such as timeouts, rate limits, and server errors: closed ↓ retryable failures reach threshold ↓ open ├─ cooldown not expired → request skipped │ └─ cooldown expired + next request ↓ half-open ├─ probe succeeds → closed └─ probe fails → open The circuit does not move to half-open automatically when the timer expires. That transition happens when the next request calls allow request after the cooldown. While the circuit is open, the recovery plan can skip that route instead of repeatedly sending requests that are likely to fail. The breaker is attached to a route rather than necessarily disabling an entire provider. One model or endpoint may be degraded while another route from the same provider remains usable. A resilient request may eventually succeed even when several things went wrong first. For example: OpenAI attempt 1: timeout OpenAI attempt 2: timeout Anthropic attempt 1: success From the application’s perspective, the request succeeded. Operationally, however, the primary route failed twice and the backup provider handled the request. That affects reliability, latency, and cost. llm-api-resilience exposes attempt metadata so the recovery path remains visible: response = llm.chat messages, max tokens=128, for attempt in response.attempts: print attempt.route name, attempt.provider, attempt.model, attempt.duration s, attempt.success, attempt.error type, An output could look like this: openai-primary openai gpt-5.6-sol 2.41 False LLMAPITimeoutError openai-primary openai gpt-5.6-sol 2.39 False LLMAPITimeoutError anthropic-backup anthropic claude-sonnet-5 1.12 True None The attempt records also contain start time and error information. They do not include a separate retry number or retry count; the final selected route is available on response.selected route . Because error message is currently derived from str error , it may contain sensitive details from an upstream exception. Applications should sanitize or restrict error messages before exporting attempt metadata to logs or external observability systems. This makes it possible to detect: The goal is not only to eventually receive a response. It is also to understand how that response was produced. Request-level failover is relatively straightforward when nothing has happened between model calls. A tool-calling session introduces a more dangerous failure mode. Imagine this sequence: user asks the agent to report an incident ↓ model requests create incident ticket ↓ application creates ticket 12345 ↓ tool result is returned ↓ model continuation fails The external side effect has already happened. The ticket exists. The application cannot safely pretend that the turn never started. Restarting the entire turn may make the model request the same tool again and create ticket 12346 for the same incident. Switching providers does not automatically solve this. The resilience layer must rebuild the provider-neutral session from the checkpoint captured before the first tool round and replay the result of the tool that already executed. This is where ordinary request fallback stops being sufficient. A resilient tool-calling session creates a checkpoint before the external tool is executed. The application then executes the tool and passes its result back through continue with . If the continuation fails with a retryable error, the library can restore the checkpoint, select a backup route, and reuse the stored tool result instead of executing the side effect again. tool call requested ↓ checkpoint created ↓ tool executed ↓ result stored in journal ↓ continuation fails ↓ backup route selected ↓ checkpoint restored ↓ stored result replayed ↓ session continues Here is a simplified incident-ticket example: python import json from llm api adapter.models.tools import ToolSpec from llm api resilience import ToolResult tools = ToolSpec name="create incident ticket", description="Create an incident ticket.", json schema={ "type": "object", "properties": { "incident id": {"type": "string"}, "severity": {"type": "string"}, }, "required": "incident id", "severity" , "additionalProperties": False, }, session = llm.session { "role": "user", "content": "Create a high-severity ticket for INC-42.", } , tools=tools, tool choice="auto", max tokens=256, first response = session.start if first response.tool calls: The checkpoint exists before the external side effect. assert session.checkpoint is not None tool results = for tool call in first response.tool calls: The application performs the external action. ticket id = f"TICKET-{tool call.arguments 'incident id' }" tool value = { "ticket id": ticket id, "status": "created", } tool results.append ToolResult tool call id=tool call.call id or tool call.name, content=json.dumps tool value , idempotency key= f"incident-ticket:" f"{tool call.arguments 'incident id' }" , replay policy="side effecting", recovered response = session.continue with tool results print "Final route:", recovered response.selected route print "Answer:", recovered response.content assert session.checkpoint is not None assert len session.journal.entries == 1 There is no public restore checkpoint call in the application code. Recovery happens inside continue with . If the primary continuation fails, the session can restore its checkpoint and continue through the next route. When the backup route produces a replay-compatible tool call, the journal returns the stored result and the application does not intentionally execute the side effect again. If the backup model changes the tool or its arguments, the application receives a new tool call that requires its own decision. This requires more than an HTTP retry wrapper. The checkpoint itself preserves: Provider-specific previous response state is intentionally excluded because it may only be compatible with the same provider and model. The tool journal separately preserves: A backup provider should receive enough normalized context to continue the session without requiring provider-specific recovery code in the application. Checkpoint recovery has an important boundary. It prevents the resilience layer from deliberately calling a completed tool again during session recovery. It does not automatically make every external system idempotent. Consider a different failure: application sends request to create ticket ↓ external service creates the ticket ↓ network fails before the response returns The application does not know whether the operation succeeded. There is no confirmed tool result to store or replay. A local checkpoint cannot resolve that uncertainty by itself. Important side effects should still use mechanisms such as: Checkpoint recovery and external idempotency solve related but different problems. Checkpoint recovery protects progress after the tool result is known. Idempotency protects the external operation when its execution status is uncertain. Real provider failures are unpredictable and expensive to reproduce. A resilience library needs deterministic tests that control exactly when every route succeeds or fails. For example, a test can define this sequence: primary start: tool call ↓ tool executes once ↓ primary continuation: timeout ↓ backup start: reconstructed tool call ↓ backup continuation: success SequenceAdapter is a test helper that returns a predefined sequence of responses and exceptions. The important assertions are not limited to the final answer: final response = session.continue with result assert final response.selected route == "backup" assert final response.content == "Ticket created successfully." assert attempt.route name for attempt in session.attempts == "primary", "primary", "backup", "backup", assert len session.journal.entries == 1 assert session.checkpoint is not None These assertions verify that the backup route completed the session, all route attempts were recorded, one tool result was journaled, and the checkpoint still exists. They do not by themselves prove that the external side effect ran only once. That requires an explicit execution counter around the tool implementation: python executions = def create ticket arguments : ticket id = f"TICKET-{arguments 'incident id' }" executions.append ticket id return { "ticket id": ticket id, "status": "created", } After recovery: assert executions == "TICKET-INC-42" With that assertion, the test proves that recovery reused the stored result without executing the external action again. Deterministic adapters are also useful for testing retries and circuit breakers. Tests should control error order and route behavior rather than hoping that a real provider fails at the right moment. Managed LLM gateways are useful. They can centralize routing, credentials, quotas, observability, billing, and organization-wide policies. llm-api-resilience targets a different boundary: the Python application itself. A library-level approach can be useful when the application needs to: A generic gateway can route inference requests, but it may not understand the application’s checkpoint, tool journal, or whether an external side effect has already happened. This does not mean that a gateway and an application-level resilience library are mutually exclusive. A gateway can manage infrastructure-level routing while the application manages stateful recovery. A resilience layer does not make different providers interchangeable in every way. Different models may interpret the same conversation differently. A backup route may complete the request but produce another style, tool choice, or decision. Failover improves availability. It does not make models deterministic substitutes. Process-local checkpoints are not sufficient for every deployment. Distributed workers, process restarts, and long-running sessions may require persistent checkpoint storage and coordination between instances. The journal prevents deliberate replay of a known completed tool result. It cannot guarantee how an external service behaves when the execution result is unknown. Retries and fallback increase latency and token usage. Policies need explicit limits rather than unlimited attempts. Providers add new error types and change response behavior. Normalized error mappings need ongoing tests and updates. These are not reasons to avoid resilience. They define where a general recovery layer ends and application-specific reliability work begins. The main benefit of building on top of a provider adapter is not only shorter configuration. It is avoiding duplicated architecture. Without a shared provider boundary, retries, circuit breakers, observability, failover, and session recovery must be integrated separately into every provider implementation. With a unified adapter contract, the application can define one recovery system across OpenAI, Anthropic, and Google. one request interface ↓ one normalized error model ↓ one recovery plan ↓ multiple providers Provider fallback protects the next LLM call. Checkpoint recovery protects the work that already happened before it. llm-api-resilience llm-api-adapter