{"slug": "multi-provider-llm-resilience-in-python-without-provider-specific-code", "title": "Multi-provider LLM resilience in Python without provider-specific code", "summary": "A developer created llm-api-resilience, a Python library that provides retries, ordered failover, circuit breakers, and checkpoint recovery across multiple LLM providers without provider-specific code. The library builds on llm-api-adapter to normalize differences between OpenAI, Anthropic, and Google APIs, allowing resilience logic to be defined once and applied to any provider.", "body_md": "OpenAI, Anthropic, and Google expose different APIs, message formats, tool-calling conventions, error types, and response structures.\n\nThat difference is manageable while an application uses only one provider. It becomes more expensive when the application needs retries, circuit breakers, fallback routes, observability, and recovery across several providers.\n\nWithout a shared abstraction, resilience logic tends to be implemented repeatedly:\n\n```\nOpenAI integration\n├── request conversion\n├── retry logic\n├── error classification\n├── circuit breaker\n└── tool-call recovery\n\nAnthropic integration\n├── request conversion\n├── retry logic\n├── error classification\n├── circuit breaker\n└── tool-call recovery\n\nGoogle integration\n├── request conversion\n├── retry logic\n├── error classification\n├── circuit breaker\n└── tool-call recovery\n```\n\nI did not want to build resilience three times.\n\nI wanted to define the recovery policy once and apply it across every provider supported by the application.\n\nThat became `llm-api-resilience`\n\n, a Python library for retries, ordered failover, circuit breakers, and checkpoint recovery across multiple LLM providers.\n\n```\npip install llm-api-resilience\n```\n\nOnce the provider adapters are configured, an ordered fallback plan takes only a few lines:\n\n``` python\nfrom llm_api_resilience import RecoveryPlan, ResilientLLM, Route\n\nllm = ResilientLLM(\n    RecoveryPlan(\n        [\n            Route(\"openai-primary\", openai_adapter),\n            Route(\"anthropic-backup\", anthropic_adapter),\n            Route(\"google-last-resort\", google_adapter),\n        ]\n    )\n)\n\nresponse = llm.chat(\n    [{\"role\": \"user\", \"content\": \"Explain circuit breakers briefly.\"}]\n)\n\nprint(response.selected_route)\nprint(response.content)\n```\n\n`llm-api-resilience`\n\n`llm-api-adapter`\n\nThe library is built on top of [ llm-api-adapter](https://github.com/Inozem/llm_api_adapter), which provides one interface for OpenAI, Anthropic, and Google.\n\nBecause provider-specific differences are handled by the adapter, the resilience layer can operate on a shared contract instead of understanding three separate APIs.\n\nThe result is a Python library with:\n\nBefore adding resilience, I had already built `llm-api-adapter`\n\nto normalize the main differences between provider APIs.\n\nThe application can create adapters for different providers and call the same `chat()`\n\nmethod:\n\n``` python\nimport os\n\nfrom llm_api_adapter.universal_adapter import UniversalLLMAPIAdapter\n\nopenai_adapter = UniversalLLMAPIAdapter(\n    organization=\"openai\",\n    model=\"gpt-5.6-sol\",\n    api_key=os.environ[\"OPENAI_API_KEY\"],\n)\n\nanthropic_adapter = UniversalLLMAPIAdapter(\n    organization=\"anthropic\",\n    model=\"claude-sonnet-5\",\n    api_key=os.environ[\"ANTHROPIC_API_KEY\"],\n)\n\ngoogle_adapter = UniversalLLMAPIAdapter(\n    organization=\"google\",\n    model=\"gemini-3.5-flash\",\n    api_key=os.environ[\"GOOGLE_API_KEY\"],\n)\n\nmessages = [\n    {\n        \"role\": \"user\",\n        \"content\": \"Reply with exactly: OK\",\n    }\n]\n\nadapters = {\n    \"openai\": openai_adapter,\n    \"anthropic\": anthropic_adapter,\n    \"google\": google_adapter,\n}\n\nfor provider, adapter in adapters.items():\n    response = adapter.chat(\n        messages=messages,\n        max_tokens=64,\n    )\n\n    print(\n        provider,\n        adapter.model,\n        \"->\",\n        response.content,\n    )\n```\n\nThe important part is not only that the method names are the same.\n\nThe adapter creates a stable boundary around:\n\nThe resilience layer does not need to know how Anthropic represents a tool result, which OpenAI parameter controls structured output, or how Google formats its request.\n\nIt only needs to understand the adapter contract.\n\nOnce that contract existed, models from different providers could become routes in the same recovery plan.\n\nA recovery plan contains an ordered collection of routes.\n\nEach route contains an adapter, a `RoutePolicy`\n\nfor retries, timeouts, and backoff, and an optional `CircuitBreaker`\n\n.\n\n``` python\nfrom llm_api_resilience import (\n    CircuitBreaker,\n    RecoveryPlan,\n    ResilientLLM,\n    Route,\n    RoutePolicy,\n)\n\nprimary_breaker = CircuitBreaker(\n    failure_threshold=3,\n    cooldown_s=30,\n)\n\nllm = ResilientLLM(\n    RecoveryPlan(\n        [\n            Route(\n                \"openai-primary\",\n                openai_adapter,\n                policy=RoutePolicy(\n                    timeout_s=15,\n                    max_attempts=2,\n                    backoff_s=0.25,\n                ),\n                breaker=primary_breaker,\n            ),\n            Route(\n                \"anthropic-backup\",\n                anthropic_adapter,\n                policy=RoutePolicy(\n                    timeout_s=20,\n                    max_attempts=1,\n                ),\n            ),\n            Route(\n                \"google-last-resort\",\n                google_adapter,\n                policy=RoutePolicy(\n                    timeout_s=30,\n                    max_attempts=1,\n                ),\n            ),\n        ]\n    )\n)\n\nresponse = llm.chat(\n    messages,\n    max_tokens=128,\n)\n\nprint(\"Selected route:\", response.selected_route)\nprint(\"Response:\", response.content)\n```\n\nThe application sends one normalized request.\n\nThe adapter handles provider-specific integration. The resilience layer decides which route to use and what to do when it fails.\n\n```\nApplication\n     ↓\nllm-api-resilience\n     ↓\nllm-api-adapter\n     ↓\nOpenAI / Anthropic / Google\n```\n\nThe application does not need separate fallback branches that rebuild messages, tools, and parameters for every provider.\n\nAdding a retry loop around every exception is easy.\n\nDeciding which failures should actually be retried is harder.\n\nA timeout may succeed on the next attempt. A server error may disappear after a short delay. A rate limit may require backoff or a switch to another route.\n\nAn invalid API key will not be fixed by retrying. Neither will an invalid schema or an incorrectly configured tool.\n\nIn the current default classifier, only normalized timeout, rate-limit, and server errors are retryable:\n\n```\nfrom llm_api_adapter.errors import (\n    LLMAPIRateLimitError,\n    LLMAPIServerError,\n    LLMAPITimeoutError,\n)\n\nfrom llm_api_resilience import DefaultFailureClassifier\n\nclassifier = DefaultFailureClassifier()\n\nprint(\n    [\n        error_type.__name__\n        for error_type in classifier.retryable_errors\n    ]\n)\n\n# [\n#     \"LLMAPITimeoutError\",\n#     \"LLMAPIRateLimitError\",\n#     \"LLMAPIServerError\",\n# ]\n```\n\nAuthentication, configuration, invalid tool or schema errors, and ordinary `ValueError`\n\nfailures are not retried by default.\n\nThis is another benefit of placing resilience above a provider adapter.\n\nEach provider exposes different exceptions, but the recovery layer does not need three unrelated classification systems. Provider-specific failures are mapped into shared error types before the recovery policy evaluates them.\n\nA route can receive multiple attempts before the recovery plan selects the next route.\n\n```\nprimary route\n      ↓\nfirst attempt fails\n      ↓\nretry policy allows another attempt\n      ↓\nretry fails\n      ↓\nroute is exhausted\n      ↓\nbackup route selected\n```\n\nFailover is ordered rather than random.\n\nThis allows the application to express preferences such as:\n\nBut every attempt has a cost.\n\nRetries add latency and token usage. A backup model may have a different price, context window, or behavioral profile.\n\nA resilience policy should therefore not mean “keep trying until something works.” It should define how much latency, cost, and model variation the application is willing to accept.\n\nRetries help with isolated errors. They are less useful when a route is already degraded.\n\nWithout a circuit breaker, every new request may wait for the same route to fail several times before moving to a healthy provider.\n\nA circuit breaker remembers recent route health and increments its failure count only for retryable errors such as timeouts, rate limits, and server errors:\n\n```\nclosed\n   ↓\nretryable failures reach threshold\n   ↓\nopen\n   ├─ cooldown not expired → request skipped\n   │\n   └─ cooldown expired + next request\n          ↓\n      half-open\n       ├─ probe succeeds → closed\n       └─ probe fails    → open\n```\n\nThe circuit does not move to half-open automatically when the timer expires. That transition happens when the next request calls `allow_request()`\n\nafter the cooldown.\n\nWhile the circuit is open, the recovery plan can skip that route instead of repeatedly sending requests that are likely to fail.\n\nThe breaker is attached to a route rather than necessarily disabling an entire provider. One model or endpoint may be degraded while another route from the same provider remains usable.\n\nA resilient request may eventually succeed even when several things went wrong first.\n\nFor example:\n\n```\nOpenAI attempt 1: timeout\nOpenAI attempt 2: timeout\nAnthropic attempt 1: success\n```\n\nFrom the application’s perspective, the request succeeded.\n\nOperationally, however, the primary route failed twice and the backup provider handled the request. That affects reliability, latency, and cost.\n\n`llm-api-resilience`\n\nexposes attempt metadata so the recovery path remains visible:\n\n```\nresponse = llm.chat(\n    messages,\n    max_tokens=128,\n)\n\nfor attempt in response.attempts:\n    print(\n        attempt.route_name,\n        attempt.provider,\n        attempt.model,\n        attempt.duration_s,\n        attempt.success,\n        attempt.error_type,\n    )\n```\n\nAn output could look like this:\n\n```\nopenai-primary     openai     gpt-5.6-sol       2.41  False  LLMAPITimeoutError\nopenai-primary     openai     gpt-5.6-sol       2.39  False  LLMAPITimeoutError\nanthropic-backup   anthropic  claude-sonnet-5   1.12  True   None\n```\n\nThe attempt records also contain start time and error information. They do not include a separate retry number or retry count; the final selected route is available on `response.selected_route`\n\n.\n\nBecause `error_message`\n\nis currently derived from `str(error)`\n\n, it may contain sensitive details from an upstream exception. Applications should sanitize or restrict error messages before exporting attempt metadata to logs or external observability systems.\n\nThis makes it possible to detect:\n\nThe goal is not only to eventually receive a response. It is also to understand how that response was produced.\n\nRequest-level failover is relatively straightforward when nothing has happened between model calls.\n\nA tool-calling session introduces a more dangerous failure mode.\n\nImagine this sequence:\n\n```\nuser asks the agent to report an incident\n        ↓\nmodel requests create_incident_ticket\n        ↓\napplication creates ticket #12345\n        ↓\ntool result is returned\n        ↓\nmodel continuation fails\n```\n\nThe external side effect has already happened.\n\nThe ticket exists. The application cannot safely pretend that the turn never started.\n\nRestarting the entire turn may make the model request the same tool again and create ticket `#12346`\n\nfor the same incident.\n\nSwitching providers does not automatically solve this. The resilience layer must rebuild the provider-neutral session from the checkpoint captured before the first tool round and replay the result of the tool that already executed.\n\nThis is where ordinary request fallback stops being sufficient.\n\nA resilient tool-calling session creates a checkpoint before the external tool is executed.\n\nThe application then executes the tool and passes its result back through `continue_with()`\n\n.\n\nIf the continuation fails with a retryable error, the library can restore the checkpoint, select a backup route, and reuse the stored tool result instead of executing the side effect again.\n\n```\ntool call requested\n        ↓\ncheckpoint created\n        ↓\ntool executed\n        ↓\nresult stored in journal\n        ↓\ncontinuation fails\n        ↓\nbackup route selected\n        ↓\ncheckpoint restored\n        ↓\nstored result replayed\n        ↓\nsession continues\n```\n\nHere is a simplified incident-ticket example:\n\n``` python\nimport json\n\nfrom llm_api_adapter.models.tools import ToolSpec\nfrom llm_api_resilience import ToolResult\n\ntools = [\n    ToolSpec(\n        name=\"create_incident_ticket\",\n        description=\"Create an incident ticket.\",\n        json_schema={\n            \"type\": \"object\",\n            \"properties\": {\n                \"incident_id\": {\"type\": \"string\"},\n                \"severity\": {\"type\": \"string\"},\n            },\n            \"required\": [\"incident_id\", \"severity\"],\n            \"additionalProperties\": False,\n        },\n    )\n]\n\nsession = llm.session(\n    [\n        {\n            \"role\": \"user\",\n            \"content\": \"Create a high-severity ticket for INC-42.\",\n        }\n    ],\n    tools=tools,\n    tool_choice=\"auto\",\n    max_tokens=256,\n)\n\nfirst_response = session.start()\n\nif first_response.tool_calls:\n    # The checkpoint exists before the external side effect.\n    assert session.checkpoint is not None\n\n    tool_results = []\n\n    for tool_call in first_response.tool_calls:\n        # The application performs the external action.\n        ticket_id = f\"TICKET-{tool_call.arguments['incident_id']}\"\n\n        tool_value = {\n            \"ticket_id\": ticket_id,\n            \"status\": \"created\",\n        }\n\n        tool_results.append(\n            ToolResult(\n                tool_call_id=tool_call.call_id or tool_call.name,\n                content=json.dumps(tool_value),\n                idempotency_key=(\n                    f\"incident-ticket:\"\n                    f\"{tool_call.arguments['incident_id']}\"\n                ),\n                replay_policy=\"side_effecting\",\n            )\n        )\n\n    recovered_response = session.continue_with(tool_results)\n\n    print(\"Final route:\", recovered_response.selected_route)\n    print(\"Answer:\", recovered_response.content)\n\n    assert session.checkpoint is not None\n    assert len(session.journal.entries) == 1\n```\n\nThere is no public `restore_checkpoint()`\n\ncall in the application code.\n\nRecovery happens inside `continue_with()`\n\n.\n\nIf the primary continuation fails, the session can restore its checkpoint and continue through the next route. When the backup route produces a replay-compatible tool call, the journal returns the stored result and the application does not intentionally execute the side effect again. If the backup model changes the tool or its arguments, the application receives a new tool call that requires its own decision.\n\nThis requires more than an HTTP retry wrapper.\n\nThe checkpoint itself preserves:\n\nProvider-specific `previous_response`\n\nstate is intentionally excluded because it may only be compatible with the same provider and model.\n\nThe tool journal separately preserves:\n\nA backup provider should receive enough normalized context to continue the session without requiring provider-specific recovery code in the application.\n\nCheckpoint recovery has an important boundary.\n\nIt prevents the resilience layer from deliberately calling a completed tool again during session recovery. It does not automatically make every external system idempotent.\n\nConsider a different failure:\n\n```\napplication sends request to create ticket\n        ↓\nexternal service creates the ticket\n        ↓\nnetwork fails before the response returns\n```\n\nThe application does not know whether the operation succeeded.\n\nThere is no confirmed tool result to store or replay.\n\nA local checkpoint cannot resolve that uncertainty by itself.\n\nImportant side effects should still use mechanisms such as:\n\nCheckpoint recovery and external idempotency solve related but different problems.\n\nCheckpoint recovery protects progress after the tool result is known.\n\nIdempotency protects the external operation when its execution status is uncertain.\n\nReal provider failures are unpredictable and expensive to reproduce.\n\nA resilience library needs deterministic tests that control exactly when every route succeeds or fails.\n\nFor example, a test can define this sequence:\n\n```\nprimary start: tool call\n        ↓\ntool executes once\n        ↓\nprimary continuation: timeout\n        ↓\nbackup start: reconstructed tool call\n        ↓\nbackup continuation: success\n```\n\n`SequenceAdapter`\n\nis a test helper that returns a predefined sequence of responses and exceptions.\n\nThe important assertions are not limited to the final answer:\n\n```\nfinal_response = session.continue_with(result)\n\nassert final_response.selected_route == \"backup\"\nassert final_response.content == \"Ticket created successfully.\"\n\nassert [\n    attempt.route_name\n    for attempt in session.attempts\n] == [\n    \"primary\",\n    \"primary\",\n    \"backup\",\n    \"backup\",\n]\n\nassert len(session.journal.entries) == 1\nassert session.checkpoint is not None\n```\n\nThese assertions verify that the backup route completed the session, all route attempts were recorded, one tool result was journaled, and the checkpoint still exists.\n\nThey do not by themselves prove that the external side effect ran only once. That requires an explicit execution counter around the tool implementation:\n\n``` python\nexecutions = []\n\ndef create_ticket(arguments):\n    ticket_id = f\"TICKET-{arguments['incident_id']}\"\n    executions.append(ticket_id)\n\n    return {\n        \"ticket_id\": ticket_id,\n        \"status\": \"created\",\n    }\n\n# After recovery:\nassert executions == [\"TICKET-INC-42\"]\n```\n\nWith that assertion, the test proves that recovery reused the stored result without executing the external action again.\n\nDeterministic adapters are also useful for testing retries and circuit breakers. Tests should control error order and route behavior rather than hoping that a real provider fails at the right moment.\n\nManaged LLM gateways are useful.\n\nThey can centralize routing, credentials, quotas, observability, billing, and organization-wide policies.\n\n`llm-api-resilience`\n\ntargets a different boundary: the Python application itself.\n\nA library-level approach can be useful when the application needs to:\n\nA generic gateway can route inference requests, but it may not understand the application’s checkpoint, tool journal, or whether an external side effect has already happened.\n\nThis does not mean that a gateway and an application-level resilience library are mutually exclusive.\n\nA gateway can manage infrastructure-level routing while the application manages stateful recovery.\n\nA resilience layer does not make different providers interchangeable in every way.\n\nDifferent models may interpret the same conversation differently.\n\nA backup route may complete the request but produce another style, tool choice, or decision.\n\nFailover improves availability. It does not make models deterministic substitutes.\n\nProcess-local checkpoints are not sufficient for every deployment.\n\nDistributed workers, process restarts, and long-running sessions may require persistent checkpoint storage and coordination between instances.\n\nThe journal prevents deliberate replay of a known completed tool result. It cannot guarantee how an external service behaves when the execution result is unknown.\n\nRetries and fallback increase latency and token usage.\n\nPolicies need explicit limits rather than unlimited attempts.\n\nProviders add new error types and change response behavior. Normalized error mappings need ongoing tests and updates.\n\nThese are not reasons to avoid resilience. They define where a general recovery layer ends and application-specific reliability work begins.\n\nThe main benefit of building on top of a provider adapter is not only shorter configuration.\n\nIt is avoiding duplicated architecture.\n\nWithout a shared provider boundary, retries, circuit breakers, observability, failover, and session recovery must be integrated separately into every provider implementation.\n\nWith a unified adapter contract, the application can define one recovery system across OpenAI, Anthropic, and Google.\n\n```\none request interface\n        ↓\none normalized error model\n        ↓\none recovery plan\n        ↓\nmultiple providers\n```\n\nProvider fallback protects the next LLM call.\n\nCheckpoint recovery protects the work that already happened before it.\n\n`llm-api-resilience`\n\n`llm-api-adapter`", "url": "https://wpnews.pro/news/multi-provider-llm-resilience-in-python-without-provider-specific-code", "canonical_source": "https://dev.to/inozem/multi-provider-llm-resilience-in-python-without-provider-specific-code-2b80", "published_at": "2026-07-22 18:31:32+00:00", "updated_at": "2026-07-22 19:00:27.431975+00:00", "lang": "en", "topics": ["developer-tools", "large-language-models", "ai-infrastructure", "artificial-intelligence"], "entities": ["OpenAI", "Anthropic", "Google", "llm-api-resilience", "llm-api-adapter"], "alternates": {"html": "https://wpnews.pro/news/multi-provider-llm-resilience-in-python-without-provider-specific-code", "markdown": "https://wpnews.pro/news/multi-provider-llm-resilience-in-python-without-provider-specific-code.md", "text": "https://wpnews.pro/news/multi-provider-llm-resilience-in-python-without-provider-specific-code.txt", "jsonld": "https://wpnews.pro/news/multi-provider-llm-resilience-in-python-without-provider-specific-code.jsonld"}}