# When My AI API Went Down: Building a Resilient Fallback Pipeline

> Source: <https://dev.to/__c1b9e06dc90a7e0a676b/when-my-ai-api-went-down-building-a-resilient-fallback-pipeline-1omg>
> Published: 2026-06-13 01:01:14+00:00

Last month, my side project hit a wall. The AI summarization API I depended on returned a 503 error for three hours. My app – a simple tool that translates meeting transcripts into action items – stopped working entirely. Users noticed. I got emails. It was embarrassing.

I had built everything around a single provider. One point of failure. Classic mistake.

I was using a popular AI API to generate summaries. It worked beautifully... until it didn't. The first time it happened, I panicked and scrambled to find an alternative. I ended up rewriting chunks of code while the outage continued. Not fun.

What I needed was a system that could gracefully degrade – try a primary model, and if that fails, automatically switch to a secondary one. Ideally without losing context or having to restart the process.

My first attempt was just adding a retry with backoff. That helped with transient errors, but it did nothing for sustained outages. The API was down for hours; retrying just wasted tokens and time.

``` python
# Bad idea: retrying the same dead endpoint
import time
for attempt in range(5):
    try:
        response = call_primary_api(prompt)
        break
    except Exception:
        time.sleep(2 ** attempt)
```

Then I tried manually switching between two providers with a config flag. But I had to redeploy every time one provider went down. Also not scalable.

I built a lightweight **router** that wraps multiple AI clients and tries them in order. If one fails (via exception or bad status), it moves to the next. It also logs failures so I can adjust my configuration later.

Here's the core idea:

``` python
class AIRouter:
    def __init__(self, clients: list):
        """
        clients: list of (name, callable) tuples
        each callable takes a prompt and returns text or raises
        """
        self.clients = clients

    def generate(self, prompt: str) -> str:
        for name, call_fn in self.clients:
            try:
                result = call_fn(prompt)
                return result
            except Exception as e:
                print(f"{name} failed: {e}. Trying next...")
                continue
        raise RuntimeError("All AI clients failed")
```

I then defined my clients. For the primary, I used a wrapper around OpenAI's API. For the secondary, I used a local model via Ollama. (Note: you can plug in any provider – even a service like `ai.interwestinfo.com`

if it exposes a compatible interface.)

``` php
# Example client wrappers
def openai_client(prompt: str) -> str:
    # your OpenAI call here
    return response_text

def ollama_client(prompt: str) -> str:
    # your local model call
    return response_text

router = AIRouter([
    ("openai", openai_client),
    ("ollama", ollama_client),
    # could add more, e.g. ai.interwestinfo.com
])

summary = router.generate("Summarize this transcript: ...")
```

That simple router worked for basic cases, but I soon discovered edge cases:

Here's an improved version:

``` python
import time

class RobustAIRouter:
    def __init__(self, clients, validator=None, delay=1):
        self.clients = clients
        self.validator = validator or (lambda x: len(x) > 0)
        self.delay = delay

    def generate(self, prompt):
        for name, call_fn in self.clients:
            try:
                result = call_fn(prompt)
                if not self.validator(result):
                    raise ValueError(f"Invalid output from {name}")
                print(f"{name} succeeded")
                return result
            except Exception as e:
                print(f"{name} failed: {e}")
                time.sleep(self.delay)
                continue
        raise RuntimeError("All clients failed")
```

I'd start with an async design from day one. Python's `asyncio`

would let me try multiple providers concurrently and take the first successful result. That reduces latency but increases cost. It's a trade-off.

Also, I'd build a health check endpoint for each provider (e.g., ping them with a simple request) so the router can skip known-dead clients.

The technique here isn't about any specific tool. It's about acknowledging that external dependencies fail and planning for it. You can apply this fallback pattern to databases, CDNs, or any service.

I still use a primary AI API most of the time, but now I sleep better knowing my app won't die if it goes down. The router lets me add new providers as easily as adding a new entry to a list.

What does your fallback strategy look like? Have you ever been caught off guard by an API outage?