Stop Letting Flaky APIs Crash Your AI Agents

A developer detailed a production-grade pattern for making AI agents resilient to flaky external APIs, combining exponential backoff, circuit breakers, and graceful fallbacks. The approach wraps every tool call in a defensive pipeline that returns degraded results with metadata, allowing the LLM to adjust its reasoning instead of crashing. The implementation uses the tenacity and circuitbreaker libraries to ensure agents survive rate limits, timeouts, and outages.

How to combine exponential backoff, circuit breakers, and graceful fallbacks for production-grade agentic workflows. AI agents are only as reliable as the tools they invoke. When an LLM decides to search the web, scrape a URL, or fetch database records, it depends entirely on network stability. In production, external APIs fail constantly. A sudden surge causes 429 rate limits, a third-party microservice throws a 504 timeout, or a target endpoint goes down entirely. The naive approach—executing raw tool calls directly inside the agent loop—is a ticking time bomb: python The Naive Anti-Pattern: Fragile Tool Execution def execute agent tool tool name: str, payload: dict : One 500 error here kills the entire multi-step reasoning chain response = requests.post f"https://api.service.internal/{tool name}", json=payload return response.json When this call breaks, the unhandled exception crashes the runtime. You lose the entire reasoning graph, waste LLM tokens, and degrade the user experience. To keep multi-step agents alive, you need a defensive execution pipeline wrapped around every tool. Instead of allowing errors to bubble up and kill the agent, we handle failures across three distinct layers: Agent Core │ ▼ ┌───────────────────────────────┐ │ Circuit Breaker Check │ │ Is Primary Service Up? │ └──────────────┬────────────────┘ OPEN │ CLOSED Healthy ┌───────┴────────┐ ▼ ▼ ┌─────────────┐ ┌───────────────────────────┐ │ Fallback │ │ Retry Engine Backoff │ │ Provider │ │ └── Primary API Endpoint │ └──────┬──────┘ └──────────────┬────────────┘ │ │ SUCCESS │ Degraded │ ▼ ▼ ┌────────────────────────────────────────────┐ │ Structured Response + Metadata │ │ LLM receives context of degradation │ └────────────────────────────────────────────┘ By returning a degraded result accompanied by metadata e.g., status: "degraded", source: "fallback cache" , the LLM can adjust its downstream reasoning rather than hallucinating over missing data. We combine tenacity for retry logic with circuitbreaker to isolate failing services. The following production-ready pattern ensures failures are caught and handled before reaching the LLM orchestrator. python from tenacity import retry, stop after attempt, wait exponential from circuitbreaker import circuit, CircuitBreakerError 1. Protect external call with circuit breaker and exponential backoff @circuit failure threshold=3, recovery timeout=60 @retry stop=stop after attempt 3 , wait=wait exponential multiplier=1, min=2, max=10 def fetch primary data query: str - dict: resp = requests.get "https://api.flaky-service.com/v1/search", params={"q": query}, timeout=3 resp.raise for status return {"data": resp.json , "source": "primary", "degraded": False} 2. Resilient fallback wrapper for the agent runtime def execute tool safely query: str - dict: try: return fetch primary data query except CircuitBreakerError, Exception as err: Fallback to internal cache or secondary tool cached result = local cache.get query or "No fresh data available." return { "data": cached result, "source": "cache fallback", "degraded": True, "error context": str err } This snippet ensures three critical guarantees: "Note: Live search is offline. Using cached results from 2 hours ago." . This keeps the model's responses