# What Happens When Your AI Feature Fails? > Source: > Published: 2026-08-19 15:30:00+00:00 The first design pass for an AI feature should describe how it fails. That can feel backwards when the feature does not work yet. But a timeout is already part of the design. So are a retrieval miss, malformed output, and a write that may have completed after the caller gave up waiting. Start with one user-visible flow. At every boundary, ask what can go wrong, what may already have happened, what the application can still promise, and how anyone will know which case occurred. Keep the answers in a failure-mode table and use it to shape the API contract, control flow, telemetry, and tests. Writing that table after the first incident is rather late. Consider a support feature that prepares a reply and optionally saves it as a draft: ``` php resolve ticket and evaluate access -> retrieve authorized ticket content and policy context -> call the model -> validate the generated reply -> save the draft when requested -> return the result ``` The sequence is easy to understand. It also leaves almost every difficult question unanswered. What does the user see when retrieval returns no policy documents? Does the model get a chance to answer anyway? If output validation fails, can the application repair it? If saving times out, is the draft absent, stored, or still being processed? Can a retry create a second draft? Which failure should wake an operator? Those are part of the feature contract. If the team postpones them until implementation, the answers tend to become whatever falls out of an exception handler or SDK default. I would rather make the awkward cases visible first. Once those decisions are written down, implementing the successful case tends to be the easy part. Before listing infrastructure failures, write down what the user believes the operation does. For the support feature, the promise might be: Prepare a reply from the ticket and approved support policy. Save it as a draft only when the user asks. Now write the outcomes the system must not present as success: This short list changes the design. Missing retrieval context cannot silently become a normal model request. A model response cannot become a persisted draft before it passes the output contract. Save success requires a confirmed write, not the absence of an exception. The unacceptable outcomes matter more than a generic goal such as "handle errors gracefully." They state which properties the application must preserve when part of the flow fails. Strictly speaking, not every row in this exercise is a system failure. An authorization denial can be correct behavior. `NoEvidence` can be a valid domain outcome. I include those adverse outcomes because they can still prevent the feature from keeping its promise, and the application needs an explicit response for them. If you already [mapped the dependencies around the model](https://dev.to/posts/the-model-is-only-one-dependency/), use that interaction map as the input. Otherwise, sketch the important calls and state changes for this one flow. Another inventory of Azure services, SDK clients, and databases will not tell you how the feature should behave. A service can fail differently in different interactions. Reading policy context may permit a reduced result. Authorizing access does not. A database read and a draft write may use the same database but have different consequences when their outcomes are unknown. Take one interaction at a time and ask: Use the category list to jog your memory; stop when more rows would describe the same effect and response. A model call can return valid JSON with an unsupported claim. A write can succeed while its acknowledgement is lost. Caller cancellation can arrive after work started. These cases are more useful than another row that simply says "dependency unavailable". For the support flow, a first pass could look like this: | Interaction and adverse condition | Effect | Side-effect state uncertainty | Application response | Signal | Test | |---|---|---|---|---|---| | Ticket authorization denies access | Protected ticket content must not be disclosed or used beyond what is necessary for the authorization decision | None | Return a generic not-found or forbidden result according to application policy | Authorization outcome and controlled resource identifier | Denied principal | | Policy retrieval returns no approved context | A grounded policy answer cannot be produced | None | Return `NoEvidence` ; do not ask the model to fill the gap | Retrieval outcome, filters, result count | Empty retrieval result | | Policy retrieval returns stale context | The reply may use obsolete rules | None | Reject the context or mark the feature temporarily unavailable | Source version and age, without document contents | Expired test document | | Model call exceeds its time budget | No reply is available inside the request budget | The provider may still be processing, but no application state changed | Stop waiting, request cancellation when supported, and return `TimedOut` ; retry behavior is decided separately | Attempt, elapsed time, cancellation reason | Delayed fake client | | Model returns malformed structured output | The reply cannot be validated | None | Reject it or make one bounded repair attempt when the contract permits | Schema version and validation category | Invalid JSON and missing fields | | Model cites a source outside the retrieved set | The reply is unsupported | None | Reject the output | Returned source IDs and validation outcome | Unknown source ID | | Draft save times out after submission | The application cannot confirm whether the draft committed, so it cannot report `Saved` | Draft may already exist | Return `Unconfirmed` with the operation ID created before submission; query or reconcile that identity before another write | Operation ID, attempt, last known state | Commit succeeds, response is dropped | | Best-effort observability export fails | Diagnosis becomes harder | Business state is unchanged | Continue and record the exporter failure locally when possible | Exporter health and dropped-item count | Disabled collector | | Required audit or security record cannot be durably written | The feature cannot satisfy its operating obligation | A protected action may already have occurred if recording is not atomic | Apply the audit policy. When the business state and record share a transaction boundary, commit them together. Otherwise prevent the action where possible or reconcile an ambiguous outcome | Audit-write outcome, policy decision, and operation ID where applicable | Rejected audit write before and after the protected action | If the cells do not affect implementation, the table is busywork. "Log the error and retry" leaves open whether retrying is safe, what the caller receives, and when the operation stops. An external effect and a local audit record do not share a transaction boundary. Record durable intent before the effect. Afterward, reconcile and record the final outcome. Depending on the operation, that recovery path may also need an idempotency key or compensation. Do not try to enumerate every exception type. Group failures when they have the same effect and response. Split them when the system must behave differently. Teams often jump from a technical symptom to a resilience mechanism: ``` php timeout -> retry invalid output -> retry dependency unavailable -> fallback ``` That skips the decision that matters: what did the failure do to the operation? A timeout on an idempotent policy read is not the same as a timeout after a draft write was submitted. Both may present as a timeout at the application boundary. The read has no side effect and might be attempted again within the remaining budget. The write has an ambiguous outcome. Retrying it with a new identity may create a duplicate. Choose the response from the effect and the known state. The exception name is only one input. For each row, I use one of a small set of response shapes: The exact set belongs to the application. Callers should receive application outcomes without having to interpret exception text. Once the failure table stabilizes, encode the outcomes that the endpoint or UI needs to handle. ``` public enum ReplyOutcome { Prepared, NoEvidence, InvalidGeneration, TimedOut, TemporarilyUnavailable } public enum DraftSaveOutcome { NotRequested, Saved, RejectedByPolicy, Failed, Unconfirmed } public sealed record PrepareReplyResult( ReplyOutcome ReplyOutcome, string? Reply, DraftSaveOutcome SaveOutcome, Guid? SaveOperationId); ``` `RejectedByPolicy` means the application deliberately refused the save before persistence. `Failed` means the application knows that persistence did not commit. `Unconfirmed` means the write may have committed, but the application has not verified the result. Allocate the operation ID before the write: ``` js Guid operationId = Guid.NewGuid(); var command = new SaveDraftCommand( OperationId: operationId, TicketId: ticketId, Reply: reply); DraftSaveResult saveResult = await draftStore.SaveAsync( command, cancellationToken); ``` The store must persist `OperationId` with the draft or an operation record and enforce uniqueness. Reusing the ID for a different command must fail. Persist enough command identity, such as the ticket ID and a canonical command fingerprint, to detect conflicting reuse. If the response disappears after commit, the application queries that identity, and any safe retry reuses it. Otherwise, the ID gives a worker nothing authoritative to reconcile. The compact result type still permits invalid combinations. Production code may use factory methods or a result hierarchy to prevent them. Even so, it makes two decisions explicit: reply generation and draft persistence have separate outcomes, and an unconfirmed save is not reported as either success or failure. The endpoint can now map those outcomes deliberately. The UI can say that a reply was prepared but its save status is still being checked. A background worker can reconcile the same operation ID. Telemetry can record stable outcome names instead of provider-specific exception messages. A full failure-mode analysis can grow quickly. Do not give every row equal attention. In a review, I start with three questions: I prefer simple priority labels such as critical, high, normal, and low over multiplying guessed scores into a number that looks scientific. Unknown write outcomes, cross-tenant data exposure, silent use of stale policy, and failures that look like success deserve attention before a clean provider error that the application already exposes honestly. Keep the treatment decision separate: mitigate, accept, or defer. A high-priority risk can still be accepted when the team understands the consequence and decides that mitigation is not justified. "We do not support draft recovery in the first release" can be a legitimate decision if the UI never claims an unconfirmed write succeeded and the consequence is acceptable. An undocumented gap is not the same thing as an accepted risk. For every important row, the team should be able to force or simulate the relevant condition and effect. Some provider and infrastructure failures cannot be reproduced exactly, but their application-visible behavior usually can. You do not need a production-scale chaos platform for the first pass. Use fake clients and controlled test doubles to return no context, delay a model call, produce malformed output, reject authorization, or lose a write acknowledgement after commit. For each high-priority row, verify four things: Keep at least the deterministic cases in the normal test suite. Run slower dependency and recovery drills on a schedule that the team will actually maintain. The same rows guide the code and tests. During an incident, they also tell the operator which behavior was intentional. The analysis should identify where another attempt may be useful. Decide retry behavior holistically across the operation afterward. That does not mean retrying the whole workflow. Whether an attempt is safe depends on idempotency, the failure class, remaining time, provider throttling, and the scope of any side effect. A retry can be a valid response to one row and make the next row much worse. A five-second model timeout tells you little on its own. It has to fit inside the request budget for retrieval, model calls, validation, tools, persistence, and any synchronous recovery attempts. Durable recovery that outlives the request needs its own deadline or operational objective. During the failure pass, note which rows need a retry or budget decision. Resist inventing a local retry count or timeout just to fill the cell. Use a failure-first pass when an AI feature reads protected data, relies on retrieval, makes authoritative claims, invokes tools, changes state, or creates a meaningful promise about latency and availability. It is also useful before changing a prompt, model, or provider when that change can alter output contracts or runtime behavior. A disposable local experiment with synthetic data and no side effects does not need a workshop or a large register. Write down the few cases that could invalidate the experiment, keep the error visible, and move on. Keep the method proportional to the feature. A small experiment needs a short list, not a reliability program. Choose one important AI flow and spend 45 minutes on its unhappy paths before adding more happy-path code. Write the user promise and the outcomes that must never appear as success. Walk each interaction, record the effect of plausible failures, and assign an application response, a diagnostic signal, and a way to force the case in a test. If a row has no defined response or cannot be tested, it is unfinished design work.