# Build your AI agent fallback routing policy before the outage hits > Source: > Published: 2026-11-09 17:27:31+00:00 *Your backup model will answer when the primary dies, and nothing will tell you it's worse. Decide now which failures you'll absorb, which you'll degrade through and which you'll refuse.* - Trip a failover only on timeouts, 5xx errors and connection failures, for example 25% failures across at least 20 requests in 60 seconds, never on 400s or single 429s. - A fallback tier earns its place only after it passes a golden set of 150 to 300 real tasks within about 3 points of the primary and emits schema-valid tool calls 99% of the time. - Prompt caches don't travel between providers, so a cold fallback can cost several times more per call; cap spend per run and per hour. - Money movement, outbound messages, production writes and anything covered by a data contract should fail closed into a queue or human handoff instead of moving to an untested model. - Pin each agent run to one model, log the model and fallback reason on every step, and rehearse the failover quarterly by blocking the primary's hostname. OpenAI's outage on December 11, 2024 took ChatGPT, the API and Sora down for hours, and the company's own postmortem traced it to a new telemetry service that overloaded its Kubernetes control plane. Teams with a second provider wired into their agents flipped the switch that afternoon. A lot of them learned the switch did more than they thought: different tool-call habits, a different bill, answers that were quietly a notch worse. An ai agent fallback routing policy is the written set of rules that decides, before anything breaks, when traffic leaves your primary model, where it goes, what it's allowed to cost and which jobs stop instead of moving. Most teams have a try/except around the client and call that a policy. This guide covers the whole thing, with numbers you can start from. They're starting values I'd test against your own traffic, not vendor standards. The position I'll argue throughout is simple. A fallback you haven't evaluated isn't a safety net. It's a second product you're launching during your worst hour, with no testing. Start with what counts as a failure, because a fallback that fires on every error is as dangerous as one that never fires. A 400 caused by a malformed request will fail on the backup too. A 429 usually means you've hit a rate limit, so slow down or move that one request. Only timeouts, 5xx responses and dropped connections say the provider itself is sick. [How to Calculate the Right AI Agent Error Budget Before You Ship](https://startupfortune.com/how-to-calculate-the-right-ai-agent-error-budget-before-you-ship/) How to calculate ai agent error budget starts with picking a target below 100% reliability, then multiplying volume by allowed error rate; a 10,000-call agent at 99% gets a 100-mistake daily budget to spend. - [how to set error budget for AI agents](https://startupfortune.com/how-to-calculate-the-right-ai-agent-error-budget-before-you-ship/) - [AI agent reliability metrics before shipping product](https://startupfortune.com/how-to-calculate-the-right-ai-agent-error-budget-before-you-ship/) A workable default for the breaker: trip it when at least 20 requests have completed in a rolling 60 seconds and 25% of them ended in a timeout, 5xx or connection error. The minimum sample matters. Without it, two failures out of three requests at 3am trips the breaker for no reason. Add a latency trigger as well, because the ugliest outages are slow ones, not loud ones. If p95 time to first token runs above three times its trailing seven-day baseline for two straight minutes, treat the provider as degraded. For per-request limits, something like 5 seconds to connect and 15 seconds to first token suits interactive agents. Batch jobs can wait longer. Once the breaker trips, keep the primary out for 60 seconds, then send it a trickle of probe traffic, say 5%, before restoring full load. Skip that half-open step and you'll flap: the primary wobbles back for ten seconds, takes everything, falls over again, and your users bounce between two models all afternoon. You don't have to build this from scratch. LiteLLM's Router exposes settings for retries, allowed failures, cooldown time and ordered fallbacks, including separate fallback lists for context-window errors and content-policy refusals. OpenRouter accepts a list of models and lets you set provider order and whether fallbacks are allowed on a request. Check each tool's current docs for exact behavior, since defaults change. Remember too that a gateway is another dependency. If it sits in front of every call, it needs its own failover story. Retries deserve one rule. Retry once on the same model with jittered backoff before failing over, but only for steps that are safe to repeat. An agent that already sent an email or created a ticket shouldn't replay that step blindly. Give every side-effecting tool an idempotency key so a replay is harmless. ## Build multi-model fallback for AI agents in tiers, and test each one Tier 1 is the same model through a different door. Anthropic's Claude models are sold through AWS Bedrock and Google Cloud's Vertex AI as well as Anthropic's own API, and OpenAI's models run on Azure OpenAI as well as OpenAI's API. Same model family, separate infrastructure and separate quotas. This is the cheapest fallback in quality terms, so build it first. Two cautions: a failure in a shared cloud region can take both doors down, and new model versions and features sometimes reach one channel before another, so pin versions and check. Tier 2 is a comparable model from a different vendor. It protects you from a vendor-wide failure, and it's where most of the surprises live. [How to Build an Eval Set for AI Agents Before You Trust One in Production](https://startupfortune.com/how-to-build-an-eval-set-for-ai-agents-before-you-trust-one-in-production/) Air Canada was ordered to pay about C$812 in February 2024 after its chatbot invented a refund policy. This guide covers where to find test cases, how to score them with code before an LLM judge, and why each case should run three to five times. - [build an eval set for AI agents](https://startupfortune.com/how-to-build-an-eval-set-for-ai-agents-before-you-trust-one-in-production/) - [AI agent test cases before production](https://startupfortune.com/how-to-build-an-eval-set-for-ai-agents-before-you-trust-one-in-production/) Tier 3 is a small, cheap model allowed to do a short list of safe things: answer from a knowledge base, summarize, triage, draft something a human will review. The restriction is the point. A small model that's allowed to do everything is a liability, while one that's allowed to do four things you've verified is a reasonable lifeboat. Tier 4 is no model at all, which we'll get to. Each tier has to pass a gate before it goes into the config. Pull 150 to 300 real tasks from production logs, attach a pass/fail check to each, and run every candidate against them. My rule of thumb: admit a model to a tier only if its task success is within about 3 points of the primary and at least 99% of its tool calls validate against your schemas. Rerun the set weekly, because providers update models under stable names and a tier that passed in September can fail in November. The tier 3 model gets a lower bar, but only on its restricted task list. Tool calling is where swaps bite hardest. Providers differ in how they format arguments, whether they emit parallel tool calls, how strictly they honor a JSON schema, how they handle call IDs and how they react when a tool returns an error. A prompt tuned for months to one model's habits will often send another down a different path, such as calling a search tool three times where the primary called it once. Keep a prompt variant per model instead of one prompt for all, and add a trace diff to your evals: same task, count the calls, compare the arguments. Don't switch models in the middle of a multi-step run if you can help it. Pin each run to one model when it starts. If that model dies at step six, restart from your last checkpoint with the new one, passing earlier tool results across as plain data. Provider-specific reasoning blocks generally don't carry over between vendors, and pasting one model's transcript into another as if it had written it invites strange behavior. ## Put a ceiling on what the backup can cost A fallback changes your bill in two ways, and the second one is the one people miss. First, price per token differs, sometimes by several times in either direction between a frontier model and a small one. Second, prompt caching doesn't travel. Both Anthropic and OpenAI discount cached input tokens, but the cache lives with the provider. An agent carrying a 20,000-token prefix of system prompt and tool definitions that hit cache all day will pay full price for that prefix on every early call to a cold backup. Add a retry storm on top and the spend line bends sharply upward during the same hours your revenue dips. So set three ceilings. A per-run budget in tokens and dollars, something like twice your normal p95 run cost, after which the run stops and escalates. A per-hour fallback spend cap, perhaps three times your trailing hourly baseline. And a priority order for shedding load when you hit it: paying customers' interactive requests first, internal tools next, batch jobs paused until the primary returns. Also check the backup's quotas. A provider account you've never sent traffic to may sit on a low rate-limit tier, and you won't find out until the failover sends it ten times its normal load. Keep a trickle of real traffic, around 1%, flowing to tier 1 and tier 2 all the time. It keeps caches warm, proves the credentials work and shows you the backup's latency before you need it. ## Know when to fail closed during an LLM API outage Degrading gracefully sounds responsible. For some jobs it's the wrong call. Fail closed on anything where a wrong action costs more than a delayed one: money movement, outbound customer messages, writes to production data, and any decision with legal, medical or compliance weight. A weaker model that drafts a refund email is a nuisance. A weaker model that issues the refund is an incident. Contracts are the other trigger, and engineers forget them. If your customer agreement promises zero data retention, a specific region or a named subprocessor list, then a fallback vendor that isn't on that list can put you in breach the moment it answers. Frankly, that's a conversation for legal before the outage, not during it. Mark those workloads as primary-only in the policy and accept the downtime. Fail closed has to mean a defined behavior, not a stack trace. Queue the job with its inputs, tell the user plainly that the assistant is paused and roughly when you'll retry, and route anything urgent to a person. The opposite failure is silent degradation: the agent keeps answering from tier 3 and nobody downstream knows. If a response came from a fallback, label it in your logs, and where it matters, tell the user. ## Write it down, log it, rehearse it Keep the policy in version-controlled config, not scattered across client code. One file should say, for each agent: the trigger thresholds, the ordered tiers, the tasks each tier may handle, the budget ceilings and the fail-closed list. A reviewer should be able to read it in five minutes and know what happens at 3am. Log the model, tier and fallback reason on every step, not just every request. Then alert on the fallback share. If more than 5% of steps have run off the primary for ten minutes, a human should know, because that's either an outage or a quiet bug in your breaker. Without per-step logs you can't tell afterward which customer outputs came from which model, and that's the first question support will ask. Then rehearse. Once a quarter, block the primary provider's hostname in staging and watch what happens. Some agents will switch cleanly. Some will fall over, which is cheap to learn in a drill. The worst ones will carry on and produce confident, slightly wrong work, and you only catch those by reading outputs, not dashboards. After staging is clean, run the same block against a small slice of production traffic during a quiet hour. Do that drill this month. The first one always finds something, and it's far better found on a Tuesday afternoon you chose than during the next provider outage, which will arrive on a day you didn't. **Also read:** [How to Calculate Rule of 40 for Startups Before Investors Do It](https://startupfortune.com/how-to-calculate-rule-of-40-for-startups-before-investors-do-it/) • [How to calculate revenue doubling time startup founders skip until a VC asks](https://startupfortune.com/how-to-calculate-revenue-doubling-time-startup-founders-skip-until-a-vc-asks/) • [How to calculate CAC payback period by channel before you scale spend](https://startupfortune.com/how-to-calculate-cac-payback-period-by-channel-before-you-scale-spend/) *This article is posted in [AI News](https://startupfortune.com/category/ai/), check it out for more related stories.* ## Join the discussion [Open in the community →](https://startupfortune.com/community/) Almost there. Sign in and your reply posts straight away.