Engineers who have supported production systems know the feeling. A dashboard turns red at two in the morning, and the first question is not just what broke but also how far will it spread. Over two decades of building financial systems, I have learned that the answer to that question is almost never decided during the incident. It was decided months earlier, in architecture reviews, when someone chose whether a dependency would be allowed to take the whole system down with it.
AI is now one of those dependencies. It is increasingly used for fraud screening, customer service, dispute handling, operational tooling, and workflows that influence financial transactions. Like any dependency, AI can fail or degrade.
What makes AI different is how it fails. Traditional services tend to fail loudly: they time out, return errors, or stop responding. AI services fail loudly too, but they also fail quietly. A model can return a fast, well-formed, confident answer that is simply wrong. From the network's point of view, nothing happened. From the business's point of view, something very important did.
Fault tolerance for AI means handling both kinds of failure. The good news is that we do not have to invent the discipline from scratch. The patterns that made distributed systems dependable (timeouts, circuit breakers, bulkheads, retries, fallbacks, graceful degradation) still apply. They just need to be extended to cover failures that look like success.
The most important design decision is a mental one. In too many architectures I review, the AI component is drawn as a solid box in the middle of the diagram, with arrows going in and out and no failure path at all. It is treated as if it were a database lookup or a pure function.
An AI component should be treated as a dependency that can fail, much like a third-party API over the public internet, with explicit limits and a tested fallback. That framing immediately changes the questions people ask:
The classic dependability literature gives us useful language here. Avižienis, Laprie, Randell, and Landwehr, in their widely cited IEEE taxonomy of dependable computing [1], separate faults, errors, and failures: a fault is the cause, an error is the incorrect internal state, and a failure occurs when the delivered service deviates from correct service. With AI, the fault is often invisible (a subtle data shift, a gap in training data, an ambiguous prompt), and the error looks like a normal response. It becomes important to stop that error before it becomes a failure.
That leads to the principle behind the rest of this article: reliability must come from the system around the model, not from the model itself.
Michael Nygard popularized circuit breakers and bulkheads in Release It! [2] nearly twenty years ago. These patterns remain useful around AI. But each needs some adjustment, because AI calls behave differently from the service calls these patterns were first designed for.
Large model inference has a wide latency spread. Response times can vary substantially with load, input size, output length, and shared computing resources. Setting a timeout based only on average latency can leave the system exposed to slow responses.
The better approach is to start from the business deadline and work backward. If a card authorization must complete in well under a second end to end, the AI step gets a fixed slice of that budget, and when the slice is spent, the system moves on without it. For generated text, I also recommend limiting output length explicitly. Limiting output length helps control latency, but the call still needs an end-to-end deadline.
A traditional circuit breaker opens when the error rate crosses a threshold. For AI, that is necessary but not enough. If a model starts returning well-formed responses that fail validation, a conventional breaker will happily stay closed, because every call "succeeded."
I think of the fix as a semantic circuit breaker. It counts validation failures, low-confidence results, and policy rejections alongside timeouts and HTTP errors. When the combined rate crosses a threshold, the breaker opens and traffic goes to the fallback path. These signals should be tracked by use case and model version so that one problematic request does not disable unrelated workflows. Recovery should use controlled probes that pass both availability and validation checks.
AI calls are slow and variable. If they share a thread pool, connection pool, or rate limit with core transaction processing, a slowdown in the model becomes a slowdown in everything. Bulkheads isolate those resources. In practice, that means separate pools for AI calls, separate quotas per use case, and a firm rule that a batch job summarizing documents can never consume the capacity reserved for real-time use cases on the critical path.
Retries need careful limits. Retrying an AI call is expensive in both latency and cost, and when a provider is overloaded, aggressive retries make the overload worse. The guidance from Google's SRE practice [3] and Amazon's Builders' Library [4] applies directly: cap attempts, use exponential backoff with jitter, and enforce a retry budget across the whole service so that retries cannot exceed a small fraction of total traffic.
Two additional rules are specific to AI. First, do not retry a response simply because you did not like it; a response that failed validation should normally go to the fallback path rather than be repeatedly sent to the same model with the same prompt. Any bounded repair attempt should fit within the same deadline and validation controls. Second, if the AI can take actions, such as calling tools or updating records, retried actions need idempotency keys or another explicit mechanism to prevent duplicate effects. A retried agent step should not update the same record or execute the same transaction twice.
The classic patterns protect us from loud failures. Quiet failures need two more mechanisms working together: validation, which reduces the risk of using an unacceptable answer, and fallback, which decides what to do when it cannot.
In financial systems, we never trust a message just because it arrived. We check its format, its signature, its amounts, and its business rules before it touches a ledger. AI output deserves the same discipline. I usually describe validation in four layers:
The first two layers are cheap and deterministic, and every production system should have them. Grounding and uncertainty checks can require additional processing, so teams should choose them based on the use case and the harm a wrong answer could cause. Passing validation does not guarantee that an answer is correct.
When the primary model times out, the breaker is open, or validation fails, the request needs somewhere to go. That destination should be designed with the same care as the primary path, not added as an afterthought.
A typical chain moves from more capable to more predictable: a smaller model with reduced scope or a cached, previously validated response that remains applicable, then a deterministic rules engine, then a human review queue or a safe hold. Each step trades capability for more constrained behavior. The chain should also be honest with downstream systems. A decision made by the rules engine should be labeled as such, so that monitoring, audit, and customer communication all know which path produced it.
A fallback model offers limited protection if it shares the primary model’s infrastructure, quotas, or other failure modes. If the primary and fallback share a failure mode, they could fail together. Independent failure modes improve resilience. Deterministic logic can provide a useful fallback when its rules remain current and appropriate for the use case.
Fallbacks can also become unavailable or reach their limits, and the AI capability may need to be switched off when its behavior is no longer acceptable. The question is what the business does then.
The answer should never be improvised. For each AI-assisted process, the best practice is to define its degradation modes in advance and write them down:
| Mode | What runs | What the customer experiences | | Full | Primary model with validation | Normal service | | Reduced | Smaller model or validated cached response, narrower scope | Slightly less personalized or slower | | Deterministic | Rules engine and static thresholds | Decisions within predefined rules and limits | | Assisted | Work queued for human review | Longer turnaround, clearly communicated | | Safe hold | Action deferred, state preserved | Request deferred; status and next steps communicated |
The most important decision in this table is one that architects cannot make alone: for each use case, should the system fail open or fail closed? A fraud model that is unavailable during a holiday peak could continue processing under approved static rules (a constrained fallback) or hold or decline transactions when the required risk assessment is unavailable. Strictly failing open would bypass the unavailable check; failing closed would block the action. A rules-based fallback sits between those choices. Neither answer is universally right. What matters is that the business chose it deliberately, before the incident, and that the choice is encoded in the system rather than left to whoever is on call.
Two practices make degradation real rather than theoretical.
For teams that already run AI in production, a short review will usually surface the biggest gaps: Fault tolerance starts with acknowledging that dependencies can fail even in a resilient system. We build redundancy because we know hardware fails, networks partition, and people make mistakes. AI makes another failure mode especially important: a component can be available, fast, and confident, and still be wrong.
The patterns we need are mostly the ones we already know. Timeouts, circuit breakers, bulkheads, retries, fallbacks, and graceful degradation remain the backbone of dependable systems. What changes is the definition of failure. Once we accept that a wrong answer is a failure even when the response code says success, those familiar patterns extend naturally to cover AI.
In my earlier article [5], I wrote that designing systems that cannot fail does not mean eliminating failure. Fault-tolerant AI is where that idea becomes concrete. The model will sometimes fail. The system should contain those failures and preserve the agreed business safeguards.
Aman Sardana is a technology leader and recognized industry expert in financial technology, payments infrastructure, and enterprise software architecture. He is recognized for his contributions to the design and modernization of large-scale financial systems, with a focus on cloud architecture, resilient platform design, and the modernization of mission-critical digital infrastructure. Aman shares insights on emerging technology trends in cloud architecture and system reliability at international technology and leadership forums. Aman holds a Master of Science in Information Technology from Northwestern University, is a Fellow of BCS, and a Senior Member of IEEE.
Disclaimer: The authors are completely responsible for the content of this article. The opinions expressed are their own and do not represent IEEE’s position nor that of the Computer Society nor its Leadership.