{"slug": "production-hardening-an-ai-video-pipeline-retries-fallbacks-and-crash-guards", "title": "Production Hardening an AI Video Pipeline: Retries, Fallbacks, and Crash Guards", "summary": "An engineer at a video generation platform detailed five reliability fixes totaling roughly 1,200 lines of code that hardened an AI pipeline against provider outages, unhandled promise rejections, and null field crashes. The changes included an error classifier with exponential backoff for transient failures, a two-tier model fallback for Claude calls, and crash guards for the Next.js server. The work addressed failures where a ten-minute backend outage exhausted all retries and a null optional field from Claude crashed plan validation.", "body_md": "The video generation platform I work on orchestrates a long chain of AI calls — a video generation provider for clip rendering, Claude for scene planning, ElevenLabs for voiceover — into finished ad creatives. When every provider was healthy, the pipeline worked. When they were not, it failed quietly: a ten-minute backend outage burned through all retries, an unhandled promise rejection took down the Next.js server with a 502 and no stack trace, and a null optional field from Claude crashed plan validation before a single clip was generated.\n\nThese were reliability gaps, not creative logic bugs. I addressed them across five focused pull requests totaling roughly 1,200 lines. This post walks through each one.\n\nThe first incident was blunt. During a live run, the video generation provider's backend went down for roughly ten minutes. All sixteen clips in the job failed. The retry logic gave up long before the outage ended.\n\nThe old configuration was simple and wrong for provider-scale outages: three retries with a fixed fifteen-second delay between attempts. That is a forty-five-second horizon. A transient backend incident routinely lasts five to fifteen minutes. Retrying three times and declaring failure is not resilience — it is giving up on the first long tail.\n\nI built an error classifier that separates transient failures from permanent ones. Transient errors — internal server errors, \"try again later\" messages, rate limits, capacity or overload signals, timeouts, HTTP 5xx, and 429 responses — get retried with exponential backoff. Non-transient errors — bad input, authentication failures, invalid parameters — fail immediately. There is no point burning retry budget on a request that will never succeed.\n\n``` js\nconst TRANSIENT_PATTERNS = [\n  /internal error/i,\n  /try again later/i,\n  /rate.?limit/i,\n  /capacity|overload/i,\n  /timeout/i,\n];\n\nfunction isTransientProviderError(msg: string, status?: number): boolean {\n  if (status === 429 || (status !== undefined && status >= 500)) return true;\n  return TRANSIENT_PATTERNS.some((re) => re.test(msg));\n}\n\nconst BACKOFF_MS = [15_000, 30_000, 60_000, 120_000];\n\nasync function generateClipWithRetry(\n  request: ClipRequest,\n  maxAttempts = BACKOFF_MS.length,\n): Promise<ClipResult> {\n  for (let attempt = 0; attempt < maxAttempts; attempt++) {\n    try {\n      return await videoProvider.generate(request);\n    } catch (err) {\n      const msg = err instanceof Error ? err.message : String(err);\n      const status = (err as { status?: number }).status;\n\n      if (!isTransientProviderError(msg, status)) throw err;\n      if (attempt === maxAttempts - 1) throw err;\n\n      await sleep(BACKOFF_MS[attempt]);\n    }\n  }\n  throw new Error('unreachable');\n}\n```\n\nThe backoff schedule spans roughly three and a half minutes per clip — enough runway to survive a mid-length provider outage. Permanent failures still surface immediately, which keeps debugging fast when the problem is on our side.\n\nThe planning stage calls Claude to produce structured scene blueprints — shot descriptions, timing markers, voiceover cues. That call was wired to a single hardcoded model with no fallback. When the primary model hit a rate limit or returned a 503, the entire job failed. There was also no visibility into cost or latency per call. I could see that planning failed; I could not see whether it failed because of tokens, latency, or a provider-side outage.\n\nI added a two-tier model strategy. The primary model comes from `ANTHROPIC_MODEL`\n\n(defaulting to `claude-opus-4-7`\n\n). When the primary fails with a transient error — HTTP 5xx, 429 — or is unavailable (404), the client automatically retries with `ANTHROPIC_FALLBACK_MODEL`\n\n. Client-side 4xx bad-request errors do not trigger fallback. If the input is malformed, switching models will not fix it.\n\n```\nfunction isTransientAnthropicError(status: number): boolean {\n  return status === 429 || status >= 500;\n}\n\nasync function callClaude(messages: Message[]): Promise<AnthropicResponse> {\n  const primary = process.env.ANTHROPIC_MODEL ?? 'claude-opus-4-7';\n  const fallback = process.env.ANTHROPIC_FALLBACK_MODEL;\n\n  const start = Date.now();\n  try {\n    const res = await anthropic.messages.create({ model: primary, messages });\n    logUsage({ model: primary, ...res.usage, latencyMs: Date.now() - start });\n    return res;\n  } catch (err) {\n    const status = (err as { status?: number }).status ?? 0;\n\n    if (!fallback || !isTransientAnthropicError(status)) throw err;\n\n    console.warn(`[anthropic] primary ${primary} failed (${status}), falling back to ${fallback}`);\n    const res = await anthropic.messages.create({ model: fallback, messages });\n    logUsage({ model: fallback, ...res.usage, latencyMs: Date.now() - start, fallback: true });\n    return res;\n  }\n}\n```\n\nEvery call now logs model name, token counts, and latency. Fallback switches are logged explicitly, which made it straightforward to correlate job failures with model outages and estimate per-job inference cost.\n\nThe most frustrating production issue was random 502 responses with nothing in the application logs. The Next.js server was dying silently. Tracing it back, a stray unhandled promise rejection in background pipeline work — fire-and-forget job updates, async callbacks without catch handlers — was terminating the Node process. In development, Node prints a warning and keeps running. In production, an unhandled rejection can crash the entire server.\n\nI added explicit process-level handlers at startup. Unhandled rejections are logged with the full stack trace and current memory usage, but the process keeps running — these are bugs worth fixing, not worth taking down every in-flight job. Uncaught exceptions are logged and followed by a clean `exit(1)`\n\n, which lets the hosting platform restart the process. Jobs auto-restore from Google Drive checkpoints, so a controlled restart is preferable to a corrupted in-memory state.\n\n``` js\nprocess.on('unhandledRejection', (reason) => {\n  console.error('[process] unhandledRejection', {\n    reason,\n    stack: reason instanceof Error ? reason.stack : undefined,\n    memory: process.memoryUsage(),\n  });\n  // Keep running — log and alert, do not crash\n});\n\nprocess.on('uncaughtException', (err) => {\n  console.error('[process] uncaughtException — exiting for clean restart', {\n    message: err.message,\n    stack: err.stack,\n    memory: process.memoryUsage(),\n  });\n  process.exit(1);\n});\n```\n\nI also removed seven dead `void updateJob;`\n\nno-ops in the regenerate route and added a `/api/health`\n\nendpoint exempted from auth middleware so Railway health checks stop getting false 401s.\n\nAfter the crash guards shipped, a different class of failure surfaced: plan validation errors. Jobs crashed with messages like `plan validation failed: scenes.3.timeMarker: Expected string, received null`\n\n. Claude was returning `null`\n\nfor fields it interpreted as \"no value.\" Our Zod schemas used `z.string().optional()`\n\n, which accepts `string`\n\nor `undefined`\n\nbut rejects `null`\n\n. This is a well-known footgun when parsing LLM JSON output.\n\nLLMs routinely emit null for absent optional fields. Zod's optional() accepts undefined but not null. Without normalization, every null is a crash.\n\nI built a reusable helper and applied it to every LLM-supplied optional string in the plan and blueprint schemas:\n\n``` js\nimport { z } from 'zod';\n\n/** Accept string | null | undefined from LLM JSON; normalize null → undefined */\nexport const llmOptionalString = z\n  .union([z.string(), z.null()])\n  .optional()\n  .transform((val) => val ?? undefined);\n\nconst sceneSchema = z.object({\n  description: z.string(),\n  timeMarker: llmOptionalString,\n  voiceoverCue: llmOptionalString,\n  cameraNote: llmOptionalString,\n});\n```\n\nThe same PR added a config-driven own-brand allow-list so the planner stops stripping the customer's own product references as \"competitor content.\"\n\nThe last issue only appeared in local end-to-end testing. Steps one through four of the pipeline passed cleanly, but every step-five clip generation failed with `image_load_error`\n\nor fetch timeouts. The video generation provider could not reach input frame URLs served through an ngrok tunnel from my laptop. Production serves frames from a stable CDN; local dev does not.\n\nThe fix is env-gated and off in production. When `UPLOAD_FRAMES_TO_KIE=1`\n\nis set, the pipeline uploads input frames to the video provider's own CDN via their File Upload API before submitting the generation request. The provider fetches from its own infrastructure — no tunnel, no timeout.\n\n```\nasync function resolveFrameUrl(localPath: string): Promise<string> {\n  if (process.env.UPLOAD_FRAMES_TO_KIE !== '1') {\n    return `${process.env.NGROK_BASE_URL}/frames/${basename(localPath)}`;\n  }\n\n  const uploaded = await videoProvider.uploadFile(localPath);\n  return uploaded.cdnUrl;\n}\n```\n\nZero production behavior change — the flag defaults off. Local developers flip one env var and run the full pipeline end-to-end without fighting tunnel networking.\n\nMulti-provider AI pipelines fail in layers, but the fix pattern is consistent: classify the error, retry or fall back only when it makes sense, log enough to diagnose the next incident, and gate dev-only workarounds behind explicit env flags.\n\nThe five PRs were deliberately small — 125 to 381 lines each — so each could be reviewed and shipped independently. Reliability work works better as targeted fixes for failure modes you actually hit in production than as a monolithic hardening sprint.", "url": "https://wpnews.pro/news/production-hardening-an-ai-video-pipeline-retries-fallbacks-and-crash-guards", "canonical_source": "https://dev.to/humzakt/production-hardening-an-ai-video-pipeline-retries-fallbacks-and-crash-guards-l9k", "published_at": "2026-08-25 20:59:33+00:00", "updated_at": "2026-08-25 21:14:20.524120+00:00", "lang": "en", "topics": ["ai-infrastructure", "ai-products", "developer-tools", "machine-learning", "artificial-intelligence"], "entities": ["Claude", "ElevenLabs", "Next.js", "Anthropic"], "alternates": {"html": "https://wpnews.pro/news/production-hardening-an-ai-video-pipeline-retries-fallbacks-and-crash-guards", "markdown": "https://wpnews.pro/news/production-hardening-an-ai-video-pipeline-retries-fallbacks-and-crash-guards.md", "text": "https://wpnews.pro/news/production-hardening-an-ai-video-pipeline-retries-fallbacks-and-crash-guards.txt", "jsonld": "https://wpnews.pro/news/production-hardening-an-ai-video-pipeline-retries-fallbacks-and-crash-guards.jsonld"}}