How to combine exponential backoff, circuit breakers, and graceful fallbacks for production-grade agentic workflows.
AI agents are only as reliable as the tools they invoke. When an LLM decides to search the web, scrape a URL, or fetch database records, it depends entirely on network stability.
In production, external APIs fail constantly. A sudden surge causes 429 rate limits, a third-party microservice throws a 504 timeout, or a target endpoint goes down entirely. The naive approach—executing raw tool calls directly inside the agent loop—is a ticking time bomb:
def execute_agent_tool(tool_name: str, payload: dict):
response = requests.post(f"https://api.service.internal/{tool_name}", json=payload)
return response.json()
When this call breaks, the unhandled exception crashes the runtime. You lose the entire reasoning graph, waste LLM tokens, and degrade the user experience.
To keep multi-step agents alive, you need a defensive execution pipeline wrapped around every tool. Instead of allowing errors to bubble up and kill the agent, we handle failures across three distinct layers:
[ Agent Core ]
│
▼
┌───────────────────────────────┐
│ Circuit Breaker Check │
│ (Is Primary Service Up?) │
└──────────────┬────────────────┘
OPEN │ CLOSED (Healthy)
┌───────┴────────┐
▼ ▼
┌─────────────┐ ┌───────────────────────────┐
│ Fallback │ │ Retry Engine (Backoff) │
│ Provider │ │ └──> Primary API Endpoint │
└──────┬──────┘ └──────────────┬────────────┘
│ │ SUCCESS
│ (Degraded) │
▼ ▼
┌────────────────────────────────────────────┐
│ Structured Response + Metadata │
│ (LLM receives context of degradation) │
└────────────────────────────────────────────┘
By returning a degraded result accompanied by metadata (e.g., status: "degraded", source: "fallback_cache"
), the LLM can adjust its downstream reasoning rather than hallucinating over missing data.
We combine tenacity
for retry logic with circuitbreaker
to isolate failing services. The following production-ready pattern ensures failures are caught and handled before reaching the LLM orchestrator.
from tenacity import retry, stop_after_attempt, wait_exponential
from circuitbreaker import circuit, CircuitBreakerError
@circuit(failure_threshold=3, recovery_timeout=60)
@retry(stop=stop_after_attempt(3), wait=wait_exponential(multiplier=1, min=2, max=10))
def fetch_primary_data(query: str) -> dict:
resp = requests.get("https://api.flaky-service.com/v1/search", params={"q": query}, timeout=3)
resp.raise_for_status()
return {"data": resp.json(), "source": "primary", "degraded": False}
def execute_tool_safely(query: str) -> dict:
try:
return fetch_primary_data(query)
except (CircuitBreakerError, Exception) as err:
cached_result = local_cache.get(query) or "No fresh data available."
return {
"data": cached_result,
"source": "cache_fallback",
"degraded": True,
"error_context": str(err)
}
This snippet ensures three critical guarantees:
"Note: Live search is offline. Using cached results from 2 hours ago."
). This keeps the model's responses