# Multi-provider LLM resilience in Python without provider-specific code

> Source: <https://dev.to/inozem/multi-provider-llm-resilience-in-python-without-provider-specific-code-2b80>
> Published: 2026-07-22 18:31:32+00:00

OpenAI, Anthropic, and Google expose different APIs, message formats, tool-calling conventions, error types, and response structures.

That difference is manageable while an application uses only one provider. It becomes more expensive when the application needs retries, circuit breakers, fallback routes, observability, and recovery across several providers.

Without a shared abstraction, resilience logic tends to be implemented repeatedly:

```
OpenAI integration
├── request conversion
├── retry logic
├── error classification
├── circuit breaker
└── tool-call recovery

Anthropic integration
├── request conversion
├── retry logic
├── error classification
├── circuit breaker
└── tool-call recovery

Google integration
├── request conversion
├── retry logic
├── error classification
├── circuit breaker
└── tool-call recovery
```

I did not want to build resilience three times.

I wanted to define the recovery policy once and apply it across every provider supported by the application.

That became `llm-api-resilience`

, a Python library for retries, ordered failover, circuit breakers, and checkpoint recovery across multiple LLM providers.

```
pip install llm-api-resilience
```

Once the provider adapters are configured, an ordered fallback plan takes only a few lines:

``` python
from llm_api_resilience import RecoveryPlan, ResilientLLM, Route

llm = ResilientLLM(
    RecoveryPlan(
        [
            Route("openai-primary", openai_adapter),
            Route("anthropic-backup", anthropic_adapter),
            Route("google-last-resort", google_adapter),
        ]
    )
)

response = llm.chat(
    [{"role": "user", "content": "Explain circuit breakers briefly."}]
)

print(response.selected_route)
print(response.content)
```

`llm-api-resilience`

`llm-api-adapter`

The library is built on top of [ llm-api-adapter](https://github.com/Inozem/llm_api_adapter), which provides one interface for OpenAI, Anthropic, and Google.

Because provider-specific differences are handled by the adapter, the resilience layer can operate on a shared contract instead of understanding three separate APIs.

The result is a Python library with:

Before adding resilience, I had already built `llm-api-adapter`

to normalize the main differences between provider APIs.

The application can create adapters for different providers and call the same `chat()`

method:

``` python
import os

from llm_api_adapter.universal_adapter import UniversalLLMAPIAdapter

openai_adapter = UniversalLLMAPIAdapter(
    organization="openai",
    model="gpt-5.6-sol",
    api_key=os.environ["OPENAI_API_KEY"],
)

anthropic_adapter = UniversalLLMAPIAdapter(
    organization="anthropic",
    model="claude-sonnet-5",
    api_key=os.environ["ANTHROPIC_API_KEY"],
)

google_adapter = UniversalLLMAPIAdapter(
    organization="google",
    model="gemini-3.5-flash",
    api_key=os.environ["GOOGLE_API_KEY"],
)

messages = [
    {
        "role": "user",
        "content": "Reply with exactly: OK",
    }
]

adapters = {
    "openai": openai_adapter,
    "anthropic": anthropic_adapter,
    "google": google_adapter,
}

for provider, adapter in adapters.items():
    response = adapter.chat(
        messages=messages,
        max_tokens=64,
    )

    print(
        provider,
        adapter.model,
        "->",
        response.content,
    )
```

The important part is not only that the method names are the same.

The adapter creates a stable boundary around:

The resilience layer does not need to know how Anthropic represents a tool result, which OpenAI parameter controls structured output, or how Google formats its request.

It only needs to understand the adapter contract.

Once that contract existed, models from different providers could become routes in the same recovery plan.

A recovery plan contains an ordered collection of routes.

Each route contains an adapter, a `RoutePolicy`

for retries, timeouts, and backoff, and an optional `CircuitBreaker`

.

``` python
from llm_api_resilience import (
    CircuitBreaker,
    RecoveryPlan,
    ResilientLLM,
    Route,
    RoutePolicy,
)

primary_breaker = CircuitBreaker(
    failure_threshold=3,
    cooldown_s=30,
)

llm = ResilientLLM(
    RecoveryPlan(
        [
            Route(
                "openai-primary",
                openai_adapter,
                policy=RoutePolicy(
                    timeout_s=15,
                    max_attempts=2,
                    backoff_s=0.25,
                ),
                breaker=primary_breaker,
            ),
            Route(
                "anthropic-backup",
                anthropic_adapter,
                policy=RoutePolicy(
                    timeout_s=20,
                    max_attempts=1,
                ),
            ),
            Route(
                "google-last-resort",
                google_adapter,
                policy=RoutePolicy(
                    timeout_s=30,
                    max_attempts=1,
                ),
            ),
        ]
    )
)

response = llm.chat(
    messages,
    max_tokens=128,
)

print("Selected route:", response.selected_route)
print("Response:", response.content)
```

The application sends one normalized request.

The adapter handles provider-specific integration. The resilience layer decides which route to use and what to do when it fails.

```
Application
     ↓
llm-api-resilience
     ↓
llm-api-adapter
     ↓
OpenAI / Anthropic / Google
```

The application does not need separate fallback branches that rebuild messages, tools, and parameters for every provider.

Adding a retry loop around every exception is easy.

Deciding which failures should actually be retried is harder.

A timeout may succeed on the next attempt. A server error may disappear after a short delay. A rate limit may require backoff or a switch to another route.

An invalid API key will not be fixed by retrying. Neither will an invalid schema or an incorrectly configured tool.

In the current default classifier, only normalized timeout, rate-limit, and server errors are retryable:

```
from llm_api_adapter.errors import (
    LLMAPIRateLimitError,
    LLMAPIServerError,
    LLMAPITimeoutError,
)

from llm_api_resilience import DefaultFailureClassifier

classifier = DefaultFailureClassifier()

print(
    [
        error_type.__name__
        for error_type in classifier.retryable_errors
    ]
)

# [
#     "LLMAPITimeoutError",
#     "LLMAPIRateLimitError",
#     "LLMAPIServerError",
# ]
```

Authentication, configuration, invalid tool or schema errors, and ordinary `ValueError`

failures are not retried by default.

This is another benefit of placing resilience above a provider adapter.

Each provider exposes different exceptions, but the recovery layer does not need three unrelated classification systems. Provider-specific failures are mapped into shared error types before the recovery policy evaluates them.

A route can receive multiple attempts before the recovery plan selects the next route.

```
primary route
      ↓
first attempt fails
      ↓
retry policy allows another attempt
      ↓
retry fails
      ↓
route is exhausted
      ↓
backup route selected
```

Failover is ordered rather than random.

This allows the application to express preferences such as:

But every attempt has a cost.

Retries add latency and token usage. A backup model may have a different price, context window, or behavioral profile.

A resilience policy should therefore not mean “keep trying until something works.” It should define how much latency, cost, and model variation the application is willing to accept.

Retries help with isolated errors. They are less useful when a route is already degraded.

Without a circuit breaker, every new request may wait for the same route to fail several times before moving to a healthy provider.

A circuit breaker remembers recent route health and increments its failure count only for retryable errors such as timeouts, rate limits, and server errors:

```
closed
   ↓
retryable failures reach threshold
   ↓
open
   ├─ cooldown not expired → request skipped
   │
   └─ cooldown expired + next request
          ↓
      half-open
       ├─ probe succeeds → closed
       └─ probe fails    → open
```

The circuit does not move to half-open automatically when the timer expires. That transition happens when the next request calls `allow_request()`

after the cooldown.

While the circuit is open, the recovery plan can skip that route instead of repeatedly sending requests that are likely to fail.

The breaker is attached to a route rather than necessarily disabling an entire provider. One model or endpoint may be degraded while another route from the same provider remains usable.

A resilient request may eventually succeed even when several things went wrong first.

For example:

```
OpenAI attempt 1: timeout
OpenAI attempt 2: timeout
Anthropic attempt 1: success
```

From the application’s perspective, the request succeeded.

Operationally, however, the primary route failed twice and the backup provider handled the request. That affects reliability, latency, and cost.

`llm-api-resilience`

exposes attempt metadata so the recovery path remains visible:

```
response = llm.chat(
    messages,
    max_tokens=128,
)

for attempt in response.attempts:
    print(
        attempt.route_name,
        attempt.provider,
        attempt.model,
        attempt.duration_s,
        attempt.success,
        attempt.error_type,
    )
```

An output could look like this:

```
openai-primary     openai     gpt-5.6-sol       2.41  False  LLMAPITimeoutError
openai-primary     openai     gpt-5.6-sol       2.39  False  LLMAPITimeoutError
anthropic-backup   anthropic  claude-sonnet-5   1.12  True   None
```

The attempt records also contain start time and error information. They do not include a separate retry number or retry count; the final selected route is available on `response.selected_route`

.

Because `error_message`

is currently derived from `str(error)`

, it may contain sensitive details from an upstream exception. Applications should sanitize or restrict error messages before exporting attempt metadata to logs or external observability systems.

This makes it possible to detect:

The goal is not only to eventually receive a response. It is also to understand how that response was produced.

Request-level failover is relatively straightforward when nothing has happened between model calls.

A tool-calling session introduces a more dangerous failure mode.

Imagine this sequence:

```
user asks the agent to report an incident
        ↓
model requests create_incident_ticket
        ↓
application creates ticket #12345
        ↓
tool result is returned
        ↓
model continuation fails
```

The external side effect has already happened.

The ticket exists. The application cannot safely pretend that the turn never started.

Restarting the entire turn may make the model request the same tool again and create ticket `#12346`

for the same incident.

Switching providers does not automatically solve this. The resilience layer must rebuild the provider-neutral session from the checkpoint captured before the first tool round and replay the result of the tool that already executed.

This is where ordinary request fallback stops being sufficient.

A resilient tool-calling session creates a checkpoint before the external tool is executed.

The application then executes the tool and passes its result back through `continue_with()`

.

If the continuation fails with a retryable error, the library can restore the checkpoint, select a backup route, and reuse the stored tool result instead of executing the side effect again.

```
tool call requested
        ↓
checkpoint created
        ↓
tool executed
        ↓
result stored in journal
        ↓
continuation fails
        ↓
backup route selected
        ↓
checkpoint restored
        ↓
stored result replayed
        ↓
session continues
```

Here is a simplified incident-ticket example:

``` python
import json

from llm_api_adapter.models.tools import ToolSpec
from llm_api_resilience import ToolResult

tools = [
    ToolSpec(
        name="create_incident_ticket",
        description="Create an incident ticket.",
        json_schema={
            "type": "object",
            "properties": {
                "incident_id": {"type": "string"},
                "severity": {"type": "string"},
            },
            "required": ["incident_id", "severity"],
            "additionalProperties": False,
        },
    )
]

session = llm.session(
    [
        {
            "role": "user",
            "content": "Create a high-severity ticket for INC-42.",
        }
    ],
    tools=tools,
    tool_choice="auto",
    max_tokens=256,
)

first_response = session.start()

if first_response.tool_calls:
    # The checkpoint exists before the external side effect.
    assert session.checkpoint is not None

    tool_results = []

    for tool_call in first_response.tool_calls:
        # The application performs the external action.
        ticket_id = f"TICKET-{tool_call.arguments['incident_id']}"

        tool_value = {
            "ticket_id": ticket_id,
            "status": "created",
        }

        tool_results.append(
            ToolResult(
                tool_call_id=tool_call.call_id or tool_call.name,
                content=json.dumps(tool_value),
                idempotency_key=(
                    f"incident-ticket:"
                    f"{tool_call.arguments['incident_id']}"
                ),
                replay_policy="side_effecting",
            )
        )

    recovered_response = session.continue_with(tool_results)

    print("Final route:", recovered_response.selected_route)
    print("Answer:", recovered_response.content)

    assert session.checkpoint is not None
    assert len(session.journal.entries) == 1
```

There is no public `restore_checkpoint()`

call in the application code.

Recovery happens inside `continue_with()`

.

If the primary continuation fails, the session can restore its checkpoint and continue through the next route. When the backup route produces a replay-compatible tool call, the journal returns the stored result and the application does not intentionally execute the side effect again. If the backup model changes the tool or its arguments, the application receives a new tool call that requires its own decision.

This requires more than an HTTP retry wrapper.

The checkpoint itself preserves:

Provider-specific `previous_response`

state is intentionally excluded because it may only be compatible with the same provider and model.

The tool journal separately preserves:

A backup provider should receive enough normalized context to continue the session without requiring provider-specific recovery code in the application.

Checkpoint recovery has an important boundary.

It prevents the resilience layer from deliberately calling a completed tool again during session recovery. It does not automatically make every external system idempotent.

Consider a different failure:

```
application sends request to create ticket
        ↓
external service creates the ticket
        ↓
network fails before the response returns
```

The application does not know whether the operation succeeded.

There is no confirmed tool result to store or replay.

A local checkpoint cannot resolve that uncertainty by itself.

Important side effects should still use mechanisms such as:

Checkpoint recovery and external idempotency solve related but different problems.

Checkpoint recovery protects progress after the tool result is known.

Idempotency protects the external operation when its execution status is uncertain.

Real provider failures are unpredictable and expensive to reproduce.

A resilience library needs deterministic tests that control exactly when every route succeeds or fails.

For example, a test can define this sequence:

```
primary start: tool call
        ↓
tool executes once
        ↓
primary continuation: timeout
        ↓
backup start: reconstructed tool call
        ↓
backup continuation: success
```

`SequenceAdapter`

is a test helper that returns a predefined sequence of responses and exceptions.

The important assertions are not limited to the final answer:

```
final_response = session.continue_with(result)

assert final_response.selected_route == "backup"
assert final_response.content == "Ticket created successfully."

assert [
    attempt.route_name
    for attempt in session.attempts
] == [
    "primary",
    "primary",
    "backup",
    "backup",
]

assert len(session.journal.entries) == 1
assert session.checkpoint is not None
```

These assertions verify that the backup route completed the session, all route attempts were recorded, one tool result was journaled, and the checkpoint still exists.

They do not by themselves prove that the external side effect ran only once. That requires an explicit execution counter around the tool implementation:

``` python
executions = []

def create_ticket(arguments):
    ticket_id = f"TICKET-{arguments['incident_id']}"
    executions.append(ticket_id)

    return {
        "ticket_id": ticket_id,
        "status": "created",
    }

# After recovery:
assert executions == ["TICKET-INC-42"]
```

With that assertion, the test proves that recovery reused the stored result without executing the external action again.

Deterministic adapters are also useful for testing retries and circuit breakers. Tests should control error order and route behavior rather than hoping that a real provider fails at the right moment.

Managed LLM gateways are useful.

They can centralize routing, credentials, quotas, observability, billing, and organization-wide policies.

`llm-api-resilience`

targets a different boundary: the Python application itself.

A library-level approach can be useful when the application needs to:

A generic gateway can route inference requests, but it may not understand the application’s checkpoint, tool journal, or whether an external side effect has already happened.

This does not mean that a gateway and an application-level resilience library are mutually exclusive.

A gateway can manage infrastructure-level routing while the application manages stateful recovery.

A resilience layer does not make different providers interchangeable in every way.

Different models may interpret the same conversation differently.

A backup route may complete the request but produce another style, tool choice, or decision.

Failover improves availability. It does not make models deterministic substitutes.

Process-local checkpoints are not sufficient for every deployment.

Distributed workers, process restarts, and long-running sessions may require persistent checkpoint storage and coordination between instances.

The journal prevents deliberate replay of a known completed tool result. It cannot guarantee how an external service behaves when the execution result is unknown.

Retries and fallback increase latency and token usage.

Policies need explicit limits rather than unlimited attempts.

Providers add new error types and change response behavior. Normalized error mappings need ongoing tests and updates.

These are not reasons to avoid resilience. They define where a general recovery layer ends and application-specific reliability work begins.

The main benefit of building on top of a provider adapter is not only shorter configuration.

It is avoiding duplicated architecture.

Without a shared provider boundary, retries, circuit breakers, observability, failover, and session recovery must be integrated separately into every provider implementation.

With a unified adapter contract, the application can define one recovery system across OpenAI, Anthropic, and Google.

```
one request interface
        ↓
one normalized error model
        ↓
one recovery plan
        ↓
multiple providers
```

Provider fallback protects the next LLM call.

Checkpoint recovery protects the work that already happened before it.

`llm-api-resilience`

`llm-api-adapter`