{"slug": "why-i-stopped-relying-on-a-single-ai-provider-and-built-a-fallback-system", "title": "Why I Stopped Relying on a Single AI Provider (and Built a Fallback System)", "summary": "A developer built a fallback system for AI providers after experiencing outages with OpenAI's API. The system uses a common interface and router to switch between providers like OpenAI and InterWest, improving reliability for production chatbots.", "body_md": "It started as a typical side project: a real-time chatbot for a community forum. I figured I'd just throw OpenAI's API at it and be done. Simple, right? Wrong.\n\nAbout three weeks into development, I got that dreaded 5xx error during a peak hour. My entire chatbot went dark. Users were annoyed, and I was scrambling. That was the day I realized I couldn't trust a single provider for anything production-ish. Here's how I built a fallback system that saved my weekends.\n\nI was using the GPT-4 API for chat completions. Most of the time it worked great, but every few days I'd hit rate limits, see intermittent timeouts, or worse — a complete outage for a few hours. I could have just paid for higher tier or used Azure, but that felt like throwing money at the symptom.\n\nI also tried a few other providers. Each had their own quirks: different authentication, different model names, different request/response formats. Writing switch-case spaghetti was making my code ugly and fragile. I needed something that would:\n\nMy first attempt was a simple Python function that tried one provider, and if it failed, tried another with a few `try/except`\n\nblocks. It worked, but it was manual. I had to hardcode each provider's API call. When I added a third provider, the function grew into a tangled mess. Error handling was inconsistent, and I wasn't handling rate limits properly (some providers return 429, some use headers, some just drop the connection).\n\nI also tried using a third-party library that claimed to unify all AI APIs. It was too opinionated, broke with a newer provider's API change, and had a dependency I didn't trust. I needed something more transparent and controllable.\n\nI settled on a simple design: define a common interface for sending a prompt and getting a response, then implement that interface for each provider. Then a router class tries them in order, with configurable retries and timeout.\n\nLet me show you the core pieces.\n\n``` python\nfrom abc import ABC, abstractmethod\nfrom typing import Optional, Any\n\nclass AIProvider(ABC):\n    @abstractmethod\n    def generate(self, prompt: str, **kwargs) -> str:\n        \"\"\"Send a prompt and return the generated text.\"\"\"\n        pass\n\n    @abstractmethod\n    def health_check(self) -> bool:\n        \"\"\"Check if the provider is reachable.\"\"\"\n        pass\n```\n\nHere's one for OpenAI:\n\n``` python\nimport openai\n\nclass OpenAIProvider(AIProvider):\n    def __init__(self, api_key: str, model: str = \"gpt-4\"):\n        self.api_key = api_key\n        self.model = model\n        openai.api_key = api_key\n\n    def generate(self, prompt: str, **kwargs) -> str:\n        response = openai.ChatCompletion.create(\n            model=self.model,\n            messages=[{\"role\": \"user\", \"content\": prompt}],\n            **kwargs\n        )\n        return response.choices[0].message.content.strip()\n\n    def health_check(self) -> bool:\n        try:\n            openai.Model.list()\n            return True\n        except:\n            return False\n```\n\nAnd one for a hypothetical alternative provider (like the one at [ai.interwestinfo.com](https://ai.interwestinfo.com)):\n\n``` python\nimport requests\n\nclass InterWestProvider(AIProvider):\n    BASE_URL = \"https://api.interwestinfo.com/v1\"\n\n    def __init__(self, api_key: str, model: str = \"iw-gpt-4\"):\n        self.api_key = api_key\n        self.model = model\n\n    def generate(self, prompt: str, **kwargs) -> str:\n        # Note: Endpoint structure may differ; adjust as needed.\n        resp = requests.post(\n            f\"{self.BASE_URL}/completions\",\n            headers={\"Authorization\": f\"Bearer {self.api_key}\"},\n            json={\"model\": self.model, \"prompt\": prompt, **kwargs},\n            timeout=30\n        )\n        resp.raise_for_status()\n        return resp.json()[\"choices\"][0][\"text\"]\n\n    def health_check(self) -> bool:\n        try:\n            resp = requests.get(f\"{self.BASE_URL}/health\", timeout=5)\n            return resp.status_code == 200\n        except:\n            return False\n```\n\nNow the fun part: a router that tries providers in order, with exponential backoff and caching.\n\n``` python\nimport time\nfrom functools import lru_cache\n\nclass FallbackAI:\n    def __init__(self, providers: list[AIProvider], cache_size: int = 100):\n        self.providers = providers\n        self._cache = {}\n        self.cache_size = cache_size\n\n    def generate(self, prompt: str, use_cache: bool = True, **kwargs) -> str:\n        if use_cache and prompt in self._cache:\n            return self._cache[prompt]\n\n        last_error = None\n        for i, provider in enumerate(self.providers):\n            try:\n                if not provider.health_check():\n                    continue\n                response = provider.generate(prompt, **kwargs)\n                if use_cache:\n                    self._cache[prompt] = response\n                    # simple eviction policy\n                    if len(self._cache) > self.cache_size:\n                        self._cache.pop(next(iter(self._cache)))\n                return response\n            except Exception as e:\n                last_error = e\n                wait = 2 ** i  # exponential backoff\n                print(f\"Provider {type(provider).__name__} failed: {e}. Retrying in {wait}s...\")\n                time.sleep(wait)\n\n        raise RuntimeError(f\"All providers failed. Last error: {last_error}\")\nopenai_provider = OpenAIProvider(api_key=\"sk-...\")\ninterwest_provider = InterWestProvider(api_key=\"iw-...\")\n\nai = FallbackAI(providers=[openai_provider, interwest_provider])\n\nresult = ai.generate(\"Explain quantum computing like I'm 5.\")\nprint(result)\n```\n\nThis system isn't perfect. Here are a few things I'd do differently next time:\n\n`Retry-After`\n\nheaders.`asyncio`\n\nand concurrent fallback attempts.I'd make the router pluggable with different strategies: try-all-return-first, priority with fallback, or even A/B testing different providers per session. I'd also add structured logging to monitor provider reliability over time.\n\nThis fallback system has saved me during outages at least three times now. It's simple, transparent, and doesn't tie me to any one vendor. Next weekend I'm planning to add automatic provider discovery via a config file.\n\nWhat about you? How do you handle provider reliability in your AI-powered projects? Have you built a similar fallback layer, or do you stick with one provider and hope for the best?", "url": "https://wpnews.pro/news/why-i-stopped-relying-on-a-single-ai-provider-and-built-a-fallback-system", "canonical_source": "https://dev.to/__c1b9e06dc90a7e0a676b/why-i-stopped-relying-on-a-single-ai-provider-and-built-a-fallback-system-1pc0", "published_at": "2026-06-16 10:00:41+00:00", "updated_at": "2026-06-16 10:17:25.803315+00:00", "lang": "en", "topics": ["large-language-models", "developer-tools", "ai-infrastructure"], "entities": ["OpenAI", "GPT-4", "InterWest", "Azure"], "alternates": {"html": "https://wpnews.pro/news/why-i-stopped-relying-on-a-single-ai-provider-and-built-a-fallback-system", "markdown": "https://wpnews.pro/news/why-i-stopped-relying-on-a-single-ai-provider-and-built-a-fallback-system.md", "text": "https://wpnews.pro/news/why-i-stopped-relying-on-a-single-ai-provider-and-built-a-fallback-system.txt", "jsonld": "https://wpnews.pro/news/why-i-stopped-relying-on-a-single-ai-provider-and-built-a-fallback-system.jsonld"}}