{"slug": "what-happens-when-your-ai-feature-fails", "title": "What Happens When Your AI Feature Fails?", "summary": "A developer argues that designing an AI feature should begin with a failure-mode analysis, not implementation. The post outlines a method for documenting adverse outcomes at every boundary, such as timeouts, retrieval misses, and uncertain writes, and using that table to shape the API contract, control flow, telemetry, and tests. The approach emphasizes defining unacceptable outcomes and user-visible promises before coding the happy path.", "body_md": "The first design pass for an AI feature should describe how it fails.\n\nThat can feel backwards when the feature does not work yet. But a timeout is already part of the design. So are a retrieval miss, malformed output, and a write that may have completed after the caller gave up waiting.\n\nStart with one user-visible flow. At every boundary, ask what can go wrong, what may already have happened, what the application can still promise, and how anyone will know which case occurred.\n\nKeep the answers in a failure-mode table and use it to shape the API contract, control flow, telemetry, and tests. Writing that table after the first incident is rather late.\n\nConsider a support feature that prepares a reply and optionally saves it as a draft:\n\n``` php\nresolve ticket and evaluate access\n    -> retrieve authorized ticket content and policy context\n    -> call the model\n    -> validate the generated reply\n    -> save the draft when requested\n    -> return the result\n```\n\nThe sequence is easy to understand. It also leaves almost every difficult question unanswered.\n\nWhat does the user see when retrieval returns no policy documents? Does the model get a chance to answer anyway? If output validation fails, can the application repair it? If saving times out, is the draft absent, stored, or still being processed? Can a retry create a second draft? Which failure should wake an operator?\n\nThose are part of the feature contract. If the team postpones them until implementation, the answers tend to become whatever falls out of an exception handler or SDK default.\n\nI would rather make the awkward cases visible first. Once those decisions are written down, implementing the successful case tends to be the easy part.\n\nBefore listing infrastructure failures, write down what the user believes the operation does.\n\nFor the support feature, the promise might be:\n\nPrepare a reply from the ticket and approved support policy. Save it as a draft only when the user asks.\n\nNow write the outcomes the system must not present as success:\n\nThis short list changes the design. Missing retrieval context cannot silently become a normal model request. A model response cannot become a persisted draft before it passes the output contract. Save success requires a confirmed write, not the absence of an exception.\n\nThe unacceptable outcomes matter more than a generic goal such as \"handle errors gracefully.\" They state which properties the application must preserve when part of the flow fails.\n\nStrictly speaking, not every row in this exercise is a system failure. An authorization denial can be correct behavior. `NoEvidence`\n\ncan be a valid domain outcome. I include those adverse outcomes because they can still prevent the feature from keeping its promise, and the application needs an explicit response for them.\n\nIf you already [mapped the dependencies around the model](https://dev.to/posts/the-model-is-only-one-dependency/), use that interaction map as the input. Otherwise, sketch the important calls and state changes for this one flow. Another inventory of Azure services, SDK clients, and databases will not tell you how the feature should behave.\n\nA service can fail differently in different interactions. Reading policy context may permit a reduced result. Authorizing access does not. A database read and a draft write may use the same database but have different consequences when their outcomes are unknown.\n\nTake one interaction at a time and ask:\n\nUse the category list to jog your memory; stop when more rows would describe the same effect and response. A model call can return valid JSON with an unsupported claim. A write can succeed while its acknowledgement is lost. Caller cancellation can arrive after work started. These cases are more useful than another row that simply says \"dependency unavailable\".\n\nFor the support flow, a first pass could look like this:\n\n| Interaction and adverse condition | Effect | Side-effect state uncertainty | Application response | Signal | Test |\n|---|---|---|---|---|---|\n| Ticket authorization denies access | Protected ticket content must not be disclosed or used beyond what is necessary for the authorization decision | None | Return a generic not-found or forbidden result according to application policy | Authorization outcome and controlled resource identifier | Denied principal |\n| Policy retrieval returns no approved context | A grounded policy answer cannot be produced | None | Return `NoEvidence` ; do not ask the model to fill the gap |\nRetrieval outcome, filters, result count | Empty retrieval result |\n| Policy retrieval returns stale context | The reply may use obsolete rules | None | Reject the context or mark the feature temporarily unavailable | Source version and age, without document contents | Expired test document |\n| Model call exceeds its time budget | No reply is available inside the request budget | The provider may still be processing, but no application state changed | Stop waiting, request cancellation when supported, and return `TimedOut` ; retry behavior is decided separately |\nAttempt, elapsed time, cancellation reason | Delayed fake client |\n| Model returns malformed structured output | The reply cannot be validated | None | Reject it or make one bounded repair attempt when the contract permits | Schema version and validation category | Invalid JSON and missing fields |\n| Model cites a source outside the retrieved set | The reply is unsupported | None | Reject the output | Returned source IDs and validation outcome | Unknown source ID |\n| Draft save times out after submission | The application cannot confirm whether the draft committed, so it cannot report `Saved`\n|\nDraft may already exist | Return `Unconfirmed` with the operation ID created before submission; query or reconcile that identity before another write |\nOperation ID, attempt, last known state | Commit succeeds, response is dropped |\n| Best-effort observability export fails | Diagnosis becomes harder | Business state is unchanged | Continue and record the exporter failure locally when possible | Exporter health and dropped-item count | Disabled collector |\n| Required audit or security record cannot be durably written | The feature cannot satisfy its operating obligation | A protected action may already have occurred if recording is not atomic | Apply the audit policy. When the business state and record share a transaction boundary, commit them together. Otherwise prevent the action where possible or reconcile an ambiguous outcome | Audit-write outcome, policy decision, and operation ID where applicable | Rejected audit write before and after the protected action |\n\nIf the cells do not affect implementation, the table is busywork. \"Log the error and retry\" leaves open whether retrying is safe, what the caller receives, and when the operation stops.\n\nAn external effect and a local audit record do not share a transaction boundary. Record durable intent before the effect. Afterward, reconcile and record the final outcome. Depending on the operation, that recovery path may also need an idempotency key or compensation.\n\nDo not try to enumerate every exception type. Group failures when they have the same effect and response. Split them when the system must behave differently.\n\nTeams often jump from a technical symptom to a resilience mechanism:\n\n``` php\ntimeout -> retry\ninvalid output -> retry\ndependency unavailable -> fallback\n```\n\nThat skips the decision that matters: what did the failure do to the operation?\n\nA timeout on an idempotent policy read is not the same as a timeout after a draft write was submitted. Both may present as a timeout at the application boundary. The read has no side effect and might be attempted again within the remaining budget. The write has an ambiguous outcome. Retrying it with a new identity may create a duplicate.\n\nChoose the response from the effect and the known state. The exception name is only one input.\n\nFor each row, I use one of a small set of response shapes:\n\nThe exact set belongs to the application. Callers should receive application outcomes without having to interpret exception text.\n\nOnce the failure table stabilizes, encode the outcomes that the endpoint or UI needs to handle.\n\n```\npublic enum ReplyOutcome\n{\n    Prepared,\n    NoEvidence,\n    InvalidGeneration,\n    TimedOut,\n    TemporarilyUnavailable\n}\n\npublic enum DraftSaveOutcome\n{\n    NotRequested,\n    Saved,\n    RejectedByPolicy,\n    Failed,\n    Unconfirmed\n}\n\npublic sealed record PrepareReplyResult(\n    ReplyOutcome ReplyOutcome,\n    string? Reply,\n    DraftSaveOutcome SaveOutcome,\n    Guid? SaveOperationId);\n```\n\n`RejectedByPolicy`\n\nmeans the application deliberately refused the save before persistence. `Failed`\n\nmeans the application knows that persistence did not commit. `Unconfirmed`\n\nmeans the write may have committed, but the application has not verified the result.\n\nAllocate the operation ID before the write:\n\n``` js\nGuid operationId = Guid.NewGuid();\n\nvar command = new SaveDraftCommand(\n    OperationId: operationId,\n    TicketId: ticketId,\n    Reply: reply);\n\nDraftSaveResult saveResult = await draftStore.SaveAsync(\n    command,\n    cancellationToken);\n```\n\nThe store must persist `OperationId`\n\nwith the draft or an operation record and enforce uniqueness. Reusing the ID for a different command must fail. Persist enough command identity, such as the ticket ID and a canonical command fingerprint, to detect conflicting reuse. If the response disappears after commit, the application queries that identity, and any safe retry reuses it. Otherwise, the ID gives a worker nothing authoritative to reconcile.\n\nThe compact result type still permits invalid combinations. Production code may use factory methods or a result hierarchy to prevent them. Even so, it makes two decisions explicit: reply generation and draft persistence have separate outcomes, and an unconfirmed save is not reported as either success or failure.\n\nThe endpoint can now map those outcomes deliberately. The UI can say that a reply was prepared but its save status is still being checked. A background worker can reconcile the same operation ID. Telemetry can record stable outcome names instead of provider-specific exception messages.\n\nA full failure-mode analysis can grow quickly. Do not give every row equal attention.\n\nIn a review, I start with three questions:\n\nI prefer simple priority labels such as critical, high, normal, and low over multiplying guessed scores into a number that looks scientific. Unknown write outcomes, cross-tenant data exposure, silent use of stale policy, and failures that look like success deserve attention before a clean provider error that the application already exposes honestly.\n\nKeep the treatment decision separate: mitigate, accept, or defer. A high-priority risk can still be accepted when the team understands the consequence and decides that mitigation is not justified. \"We do not support draft recovery in the first release\" can be a legitimate decision if the UI never claims an unconfirmed write succeeded and the consequence is acceptable. An undocumented gap is not the same thing as an accepted risk.\n\nFor every important row, the team should be able to force or simulate the relevant condition and effect. Some provider and infrastructure failures cannot be reproduced exactly, but their application-visible behavior usually can.\n\nYou do not need a production-scale chaos platform for the first pass. Use fake clients and controlled test doubles to return no context, delay a model call, produce malformed output, reject authorization, or lose a write acknowledgement after commit.\n\nFor each high-priority row, verify four things:\n\nKeep at least the deterministic cases in the normal test suite. Run slower dependency and recovery drills on a schedule that the team will actually maintain.\n\nThe same rows guide the code and tests. During an incident, they also tell the operator which behavior was intentional.\n\nThe analysis should identify where another attempt may be useful. Decide retry behavior holistically across the operation afterward. That does not mean retrying the whole workflow.\n\nWhether an attempt is safe depends on idempotency, the failure class, remaining time, provider throttling, and the scope of any side effect. A retry can be a valid response to one row and make the next row much worse.\n\nA five-second model timeout tells you little on its own. It has to fit inside the request budget for retrieval, model calls, validation, tools, persistence, and any synchronous recovery attempts. Durable recovery that outlives the request needs its own deadline or operational objective.\n\nDuring the failure pass, note which rows need a retry or budget decision. Resist inventing a local retry count or timeout just to fill the cell.\n\nUse a failure-first pass when an AI feature reads protected data, relies on retrieval, makes authoritative claims, invokes tools, changes state, or creates a meaningful promise about latency and availability. It is also useful before changing a prompt, model, or provider when that change can alter output contracts or runtime behavior.\n\nA disposable local experiment with synthetic data and no side effects does not need a workshop or a large register. Write down the few cases that could invalidate the experiment, keep the error visible, and move on.\n\nKeep the method proportional to the feature. A small experiment needs a short list, not a reliability program.\n\nChoose one important AI flow and spend 45 minutes on its unhappy paths before adding more happy-path code.\n\nWrite the user promise and the outcomes that must never appear as success. Walk each interaction, record the effect of plausible failures, and assign an application response, a diagnostic signal, and a way to force the case in a test.\n\nIf a row has no defined response or cannot be tested, it is unfinished design work.", "url": "https://wpnews.pro/news/what-happens-when-your-ai-feature-fails", "canonical_source": "https://dev.to/lukaswalter/what-happens-when-your-ai-feature-fails-4d5e", "published_at": "2026-08-19 15:30:00+00:00", "updated_at": "2026-08-19 15:43:12.415214+00:00", "lang": "en", "topics": ["artificial-intelligence", "ai-products", "developer-tools"], "entities": [], "alternates": {"html": "https://wpnews.pro/news/what-happens-when-your-ai-feature-fails", "markdown": "https://wpnews.pro/news/what-happens-when-your-ai-feature-fails.md", "text": "https://wpnews.pro/news/what-happens-when-your-ai-feature-fails.txt", "jsonld": "https://wpnews.pro/news/what-happens-when-your-ai-feature-fails.jsonld"}}