It started as a typical side project: a real-time chatbot for a community forum. I figured I'd just throw OpenAI's API at it and be done. Simple, right? Wrong.
About three weeks into development, I got that dreaded 5xx error during a peak hour. My entire chatbot went dark. Users were annoyed, and I was scrambling. That was the day I realized I couldn't trust a single provider for anything production-ish. Here's how I built a fallback system that saved my weekends.
I was using the GPT-4 API for chat completions. Most of the time it worked great, but every few days I'd hit rate limits, see intermittent timeouts, or worse — a complete outage for a few hours. I could have just paid for higher tier or used Azure, but that felt like throwing money at the symptom.
I also tried a few other providers. Each had their own quirks: different authentication, different model names, different request/response formats. Writing switch-case spaghetti was making my code ugly and fragile. I needed something that would:
My first attempt was a simple Python function that tried one provider, and if it failed, tried another with a few try/except
blocks. It worked, but it was manual. I had to hardcode each provider's API call. When I added a third provider, the function grew into a tangled mess. Error handling was inconsistent, and I wasn't handling rate limits properly (some providers return 429, some use headers, some just drop the connection).
I also tried using a third-party library that claimed to unify all AI APIs. It was too opinionated, broke with a newer provider's API change, and had a dependency I didn't trust. I needed something more transparent and controllable.
I settled on a simple design: define a common interface for sending a prompt and getting a response, then implement that interface for each provider. Then a router class tries them in order, with configurable retries and timeout.
Let me show you the core pieces.
from abc import ABC, abstractmethod
from typing import Optional, Any
class AIProvider(ABC):
@abstractmethod
def generate(self, prompt: str, **kwargs) -> str:
"""Send a prompt and return the generated text."""
pass
@abstractmethod
def health_check(self) -> bool:
"""Check if the provider is reachable."""
pass
Here's one for OpenAI:
import openai
class OpenAIProvider(AIProvider):
def __init__(self, api_key: str, model: str = "gpt-4"):
self.api_key = api_key
self.model = model
openai.api_key = api_key
def generate(self, prompt: str, **kwargs) -> str:
response = openai.ChatCompletion.create(
model=self.model,
messages=[{"role": "user", "content": prompt}],
**kwargs
)
return response.choices[0].message.content.strip()
def health_check(self) -> bool:
try:
openai.Model.list()
return True
except:
return False
And one for a hypothetical alternative provider (like the one at ai.interwestinfo.com):
import requests
class InterWestProvider(AIProvider):
BASE_URL = "https://api.interwestinfo.com/v1"
def __init__(self, api_key: str, model: str = "iw-gpt-4"):
self.api_key = api_key
self.model = model
def generate(self, prompt: str, **kwargs) -> str:
resp = requests.post(
f"{self.BASE_URL}/completions",
headers={"Authorization": f"Bearer {self.api_key}"},
json={"model": self.model, "prompt": prompt, **kwargs},
timeout=30
)
resp.raise_for_status()
return resp.json()["choices"][0]["text"]
def health_check(self) -> bool:
try:
resp = requests.get(f"{self.BASE_URL}/health", timeout=5)
return resp.status_code == 200
except:
return False
Now the fun part: a router that tries providers in order, with exponential backoff and caching.
import time
from functools import lru_cache
class FallbackAI:
def __init__(self, providers: list[AIProvider], cache_size: int = 100):
self.providers = providers
self._cache = {}
self.cache_size = cache_size
def generate(self, prompt: str, use_cache: bool = True, **kwargs) -> str:
if use_cache and prompt in self._cache:
return self._cache[prompt]
last_error = None
for i, provider in enumerate(self.providers):
try:
if not provider.health_check():
continue
response = provider.generate(prompt, **kwargs)
if use_cache:
self._cache[prompt] = response
if len(self._cache) > self.cache_size:
self._cache.pop(next(iter(self._cache)))
return response
except Exception as e:
last_error = e
wait = 2 ** i # exponential backoff
print(f"Provider {type(provider).__name__} failed: {e}. Retrying in {wait}s...")
time.sleep(wait)
raise RuntimeError(f"All providers failed. Last error: {last_error}")
openai_provider = OpenAIProvider(api_key="sk-...")
interwest_provider = InterWestProvider(api_key="iw-...")
ai = FallbackAI(providers=[openai_provider, interwest_provider])
result = ai.generate("Explain quantum computing like I'm 5.")
print(result)
This system isn't perfect. Here are a few things I'd do differently next time:
Retry-After
headers.asyncio
and concurrent fallback attempts.I'd make the router pluggable with different strategies: try-all-return-first, priority with fallback, or even A/B testing different providers per session. I'd also add structured logging to monitor provider reliability over time.
This fallback system has saved me during outages at least three times now. It's simple, transparent, and doesn't tie me to any one vendor. Next weekend I'm planning to add automatic provider discovery via a config file.
What about you? How do you handle provider reliability in your AI-powered projects? Have you built a similar fallback layer, or do you stick with one provider and hope for the best?