{"slug": "building-reliable-agentic-pipelines-retries-fallbacks-and-observability", "title": "Building reliable agentic pipelines — retries, fallbacks, and observability", "summary": "A developer building the AI chatbot for their company product, Weeve, detailed four production-hardening patterns for agentic pipelines after a load test hung for eleven minutes and then crashed. The patterns — exponential-backoff retries, fallback paths to cached data, loop detection over a sliding window of agent steps, and structured observability — address tool call failures, context overflow, silent loops, and hallucinated JSON that demos on clean inputs never surface.", "body_md": "Agent demos work. Production agents break — tool failures, context overflow, silent loops, hallucinated JSON. Four patterns that close the gap: retry, fallback, loop detection, and structured observability.\n\nThe demo worked. Flawlessly. The agent found the customer record, pulled the transaction history, summarized it, and drafted the response in under four seconds. We showed it to the team on a Friday. Monday morning, in a production load test, it hung. For eleven minutes. Then crashed.\n\nAgent demos work because they run on clean inputs, stable APIs, and a single user. Production means flaky third-party tool endpoints, context windows that overflow mid-conversation, loops that spin silently because no exit condition matched, and hallucinations that look like valid JSON until you try to reserialize them.\n\nI ran into all four building the AI chatbot for my company product, Weeve. Here is what I added to close the gap.\n\nTool call failures are the most common production failure mode. An external API returns a 503. A database query times out under load. The agent's default behaviour: propagate the error and crash the run.\n\n```\n// Generic retry wrapper -- works around any async function, not just agent tool calls\nasync function withRetry<T>(\n  fn: () => Promise<T>,\n  { maxAttempts = 3, baseDelayMs = 500 }: { maxAttempts?: number; baseDelayMs?: number } = {}\n): Promise<T> {\n  for (let attempt = 1; attempt <= maxAttempts; attempt++) {\n    try {\n      return await fn()\n    } catch (err) {\n      if (attempt === maxAttempts) throw err // Last attempt -- let it propagate\n      // Math.pow(2, attempt - 1): 1x, 2x, 4x -- backs off exponentially\n      const delay = baseDelayMs * Math.pow(2, attempt - 1)\n      await new Promise(res => setTimeout(res, delay))\n    }\n  }\n  throw new Error('unreachable') // TypeScript needs this -- the loop always returns or throws\n}\n\n// Wrap any tool call -- the agent code stays unchanged\nconst result = await withRetry(\n  () => externalCrmApi.getCustomer(customerId),\n  { maxAttempts: 3, baseDelayMs: 300 }\n)\n```\n\nDouble the delay on each retry (exponential backoff) to avoid hammering an already-struggling API. For non-idempotent operations — writes, payments — don't retry automatically. Flag them for human review instead.\n\nRetries handle transient failures. Persistent failures need a fallback path — something the agent can do when the primary tool is unavailable, rather than returning nothing.\n\n``` js\nasync function withFallback<T>(\n  primary: () => Promise<T>,\n  fallback: () => Promise<T>\n): Promise<T> {\n  try {\n    // Try primary with a quick retry budget -- don't wait too long before giving up\n    return await withRetry(primary, { maxAttempts: 2, baseDelayMs: 200 })\n  } catch {\n    // Primary exhausted its retries -- fall back instead of propagating the error\n    return await fallback()\n  }\n}\n\n// Real usage: live data -> cached snapshot when live is unavailable\nconst customerData = await withFallback(\n  () => liveApi.getCustomer(customerId),\n  () => cache.getLastKnownCustomer(customerId) // Stale but useful\n)\n```\n\nFallbacks keep the agent useful under degraded conditions. A cached snapshot is worse than live data — but it is infinitely better than a crashed run and a blank response.\n\nInfinite loops are silent in agentic systems. The agent calls a tool, gets an ambiguous result, decides it needs more information, calls the same tool again. No exception is raised. The loop spins until the context window overflows or the timeout kills it.\n\n```\ninterface AgentStep {\n  toolName: string\n  input: string\n}\n\n// windowSize: only check the last N steps -- agents can revisit tools legitimately over a long run\nfunction detectLoop(steps: AgentStep[], windowSize = 6): boolean {\n  if (steps.length < windowSize) return false\n  const recent = steps.slice(-windowSize)\n  const seen = new Set<string>()\n  for (const step of recent) {\n    // Fingerprint = tool name + serialised input -- catches exact repeats\n    const key = `${step.toolName}:${step.input}`\n    if (seen.has(key)) return true // Same call twice in the window = loop\n    seen.add(key)\n  }\n  return false\n}\n\n// Check after every step in the agent run loop\nif (detectLoop(executedSteps)) {\n  throw new Error(\n    `Agent loop detected after ${executedSteps.length} steps -- breaking run`\n  )\n}\n```\n\nFingerprint each step as tool name + serialised input. If you see the same fingerprint twice in a short window, you have a loop. Break it loudly — an error you can catch and log is better than a silent timeout.\n\nLogging 'agent failed' is not observability. Observability for an agent run means knowing: which tool was called, with what arguments, how long it took, what it returned, and what the agent decided to do next.\n\n```\ninterface AgentSpan {\n  runId: string     // Ties all spans for one agent run together\n  step: number      // Execution order within the run\n  toolName: string\n  inputSummary: string\n  durationMs: number\n  success: boolean\n  errorMessage?: string\n  outputSummary?: string\n}\n\n// Wrap every tool call with this -- don't call tools directly\nasync function tracedToolCall(\n  runId: string,\n  step: number,\n  toolName: string,\n  args: unknown,\n  fn: () => Promise<unknown>\n): Promise<unknown> {\n  const start = Date.now()\n  const inputSummary = JSON.stringify(args).slice(0, 200) // Truncate large inputs\n  try {\n    const result = await fn()\n    await observabilityStore.write({\n      runId, step, toolName, inputSummary,\n      durationMs: Date.now() - start,\n      success: true,\n      outputSummary: JSON.stringify(result).slice(0, 200),\n    } satisfies AgentSpan) // satisfies: compiler checks shape without widening the type\n    return result\n  } catch (err) {\n    await observabilityStore.write({\n      runId, step, toolName, inputSummary,\n      durationMs: Date.now() - start,\n      success: false,\n      errorMessage: err instanceof Error ? err.message : String(err),\n    } satisfies AgentSpan)\n    throw err // Re-throw so the agent run still sees the failure\n  }\n}\n```\n\nEvery tool call is a span. The runId ties all spans for a single agent run together — so when something fails, you can reconstruct exactly what happened, in order. This is the difference between debugging production and guessing about it.\n\nIn production, these four patterns compose. The agent run starts with a runId. Every tool call goes through tracedToolCall, which wraps retry and fallback internally, logs each attempt as a span, and records failures without crashing the run. After each step, the loop detector scans the last six steps. If a loop is found, it throws — which gets caught, logged as a terminal span, and returned to the caller as a structured error rather than a silent hang.\n\nThe result: agent runs that degrade gracefully, fail loudly, and leave a breadcrumb trail you can actually follow. Not the demo. The thing that runs on Monday morning.\n\n*Run all four patterns yourself — retry, fallback, loop detection, and observability wired together*", "url": "https://wpnews.pro/news/building-reliable-agentic-pipelines-retries-fallbacks-and-observability", "canonical_source": "https://dev.to/letusai15/building-reliable-agentic-pipelines-retries-fallbacks-and-observability-2kl3", "published_at": "2026-09-10 14:30:00+00:00", "updated_at": "2026-09-10 14:44:39.301723+00:00", "lang": "en", "topics": ["ai-agents", "ai-tools", "developer-tools", "mlops"], "entities": ["Weeve"], "alternates": {"html": "https://wpnews.pro/news/building-reliable-agentic-pipelines-retries-fallbacks-and-observability", "markdown": "https://wpnews.pro/news/building-reliable-agentic-pipelines-retries-fallbacks-and-observability.md", "text": "https://wpnews.pro/news/building-reliable-agentic-pipelines-retries-fallbacks-and-observability.txt", "jsonld": "https://wpnews.pro/news/building-reliable-agentic-pipelines-retries-fallbacks-and-observability.jsonld"}}