Production Hardening an AI Video Pipeline: Retries, Fallbacks, and Crash Guards

An engineer at a video generation platform detailed five reliability fixes totaling roughly 1,200 lines of code that hardened an AI pipeline against provider outages, unhandled promise rejections, and null field crashes. The changes included an error classifier with exponential backoff for transient failures, a two-tier model fallback for Claude calls, and crash guards for the Next.js server. The work addressed failures where a ten-minute backend outage exhausted all retries and a null optional field from Claude crashed plan validation.

The video generation platform I work on orchestrates a long chain of AI calls — a video generation provider for clip rendering, Claude for scene planning, ElevenLabs for voiceover — into finished ad creatives. When every provider was healthy, the pipeline worked. When they were not, it failed quietly: a ten-minute backend outage burned through all retries, an unhandled promise rejection took down the Next.js server with a 502 and no stack trace, and a null optional field from Claude crashed plan validation before a single clip was generated. These were reliability gaps, not creative logic bugs. I addressed them across five focused pull requests totaling roughly 1,200 lines. This post walks through each one. The first incident was blunt. During a live run, the video generation provider's backend went down for roughly ten minutes. All sixteen clips in the job failed. The retry logic gave up long before the outage ended. The old configuration was simple and wrong for provider-scale outages: three retries with a fixed fifteen-second delay between attempts. That is a forty-five-second horizon. A transient backend incident routinely lasts five to fifteen minutes. Retrying three times and declaring failure is not resilience — it is giving up on the first long tail. I built an error classifier that separates transient failures from permanent ones. Transient errors — internal server errors, "try again later" messages, rate limits, capacity or overload signals, timeouts, HTTP 5xx, and 429 responses — get retried with exponential backoff. Non-transient errors — bad input, authentication failures, invalid parameters — fail immediately. There is no point burning retry budget on a request that will never succeed. js const TRANSIENT PATTERNS = /internal error/i, /try again later/i, /rate.?limit/i, /capacity|overload/i, /timeout/i, ; function isTransientProviderError msg: string, status?: number : boolean { if status === 429 || status == undefined && status = 500 return true; return TRANSIENT PATTERNS.some re = re.test msg ; } const BACKOFF MS = 15 000, 30 000, 60 000, 120 000 ; async function generateClipWithRetry request: ClipRequest, maxAttempts = BACKOFF MS.length, : Promise<ClipResult { for let attempt = 0; attempt < maxAttempts; attempt++ { try { return await videoProvider.generate request ; } catch err { const msg = err instanceof Error ? err.message : String err ; const status = err as { status?: number } .status; if isTransientProviderError msg, status throw err; if attempt === maxAttempts - 1 throw err; await sleep BACKOFF MS attempt ; } } throw new Error 'unreachable' ; } The backoff schedule spans roughly three and a half minutes per clip — enough runway to survive a mid-length provider outage. Permanent failures still surface immediately, which keeps debugging fast when the problem is on our side. The planning stage calls Claude to produce structured scene blueprints — shot descriptions, timing markers, voiceover cues. That call was wired to a single hardcoded model with no fallback. When the primary model hit a rate limit or returned a 503, the entire job failed. There was also no visibility into cost or latency per call. I could see that planning failed; I could not see whether it failed because of tokens, latency, or a provider-side outage. I added a two-tier model strategy. The primary model comes from ANTHROPIC MODEL defaulting to claude-opus-4-7 . When the primary fails with a transient error — HTTP 5xx, 429 — or is unavailable 404 , the client automatically retries with ANTHROPIC FALLBACK MODEL . Client-side 4xx bad-request errors do not trigger fallback. If the input is malformed, switching models will not fix it. function isTransientAnthropicError status: number : boolean { return status === 429 || status = 500; } async function callClaude messages: Message : Promise<AnthropicResponse { const primary = process.env.ANTHROPIC MODEL ?? 'claude-opus-4-7'; const fallback = process.env.ANTHROPIC FALLBACK MODEL; const start = Date.now ; try { const res = await anthropic.messages.create { model: primary, messages } ; logUsage { model: primary, ...res.usage, latencyMs: Date.now - start } ; return res; } catch err { const status = err as { status?: number } .status ?? 0; if fallback || isTransientAnthropicError status throw err; console.warn anthropic primary ${primary} failed ${status} , falling back to ${fallback} ; const res = await anthropic.messages.create { model: fallback, messages } ; logUsage { model: fallback, ...res.usage, latencyMs: Date.now - start, fallback: true } ; return res; } } Every call now logs model name, token counts, and latency. Fallback switches are logged explicitly, which made it straightforward to correlate job failures with model outages and estimate per-job inference cost. The most frustrating production issue was random 502 responses with nothing in the application logs. The Next.js server was dying silently. Tracing it back, a stray unhandled promise rejection in background pipeline work — fire-and-forget job updates, async callbacks without catch handlers — was terminating the Node process. In development, Node prints a warning and keeps running. In production, an unhandled rejection can crash the entire server. I added explicit process-level handlers at startup. Unhandled rejections are logged with the full stack trace and current memory usage, but the process keeps running — these are bugs worth fixing, not worth taking down every in-flight job. Uncaught exceptions are logged and followed by a clean exit 1 , which lets the hosting platform restart the process. Jobs auto-restore from Google Drive checkpoints, so a controlled restart is preferable to a corrupted in-memory state. js process.on 'unhandledRejection', reason = { console.error ' process unhandledRejection', { reason, stack: reason instanceof Error ? reason.stack : undefined, memory: process.memoryUsage , } ; // Keep running — log and alert, do not crash } ; process.on 'uncaughtException', err = { console.error ' process uncaughtException — exiting for clean restart', { message: err.message, stack: err.stack, memory: process.memoryUsage , } ; process.exit 1 ; } ; I also removed seven dead void updateJob; no-ops in the regenerate route and added a /api/health endpoint exempted from auth middleware so Railway health checks stop getting false 401s. After the crash guards shipped, a different class of failure surfaced: plan validation errors. Jobs crashed with messages like plan validation failed: scenes.3.timeMarker: Expected string, received null . Claude was returning null for fields it interpreted as "no value." Our Zod schemas used z.string .optional , which accepts string or undefined but rejects null . This is a well-known footgun when parsing LLM JSON output. LLMs routinely emit null for absent optional fields. Zod's optional accepts undefined but not null. Without normalization, every null is a crash. I built a reusable helper and applied it to every LLM-supplied optional string in the plan and blueprint schemas: js import { z } from 'zod'; / Accept string | null | undefined from LLM JSON; normalize null → undefined / export const llmOptionalString = z .union z.string , z.null .optional .transform val = val ?? undefined ; const sceneSchema = z.object { description: z.string , timeMarker: llmOptionalString, voiceoverCue: llmOptionalString, cameraNote: llmOptionalString, } ; The same PR added a config-driven own-brand allow-list so the planner stops stripping the customer's own product references as "competitor content." The last issue only appeared in local end-to-end testing. Steps one through four of the pipeline passed cleanly, but every step-five clip generation failed with image load error or fetch timeouts. The video generation provider could not reach input frame URLs served through an ngrok tunnel from my laptop. Production serves frames from a stable CDN; local dev does not. The fix is env-gated and off in production. When UPLOAD FRAMES TO KIE=1 is set, the pipeline uploads input frames to the video provider's own CDN via their File Upload API before submitting the generation request. The provider fetches from its own infrastructure — no tunnel, no timeout. async function resolveFrameUrl localPath: string : Promise<string { if process.env.UPLOAD FRAMES TO KIE == '1' { return ${process.env.NGROK BASE URL}/frames/${basename localPath } ; } const uploaded = await videoProvider.uploadFile localPath ; return uploaded.cdnUrl; } Zero production behavior change — the flag defaults off. Local developers flip one env var and run the full pipeline end-to-end without fighting tunnel networking. Multi-provider AI pipelines fail in layers, but the fix pattern is consistent: classify the error, retry or fall back only when it makes sense, log enough to diagnose the next incident, and gate dev-only workarounds behind explicit env flags. The five PRs were deliberately small — 125 to 381 lines each — so each could be reviewed and shipped independently. Reliability work works better as targeted fixes for failure modes you actually hit in production than as a monolithic hardening sprint.