Building reliable agentic pipelines — retries, fallbacks, and observability

A developer building the AI chatbot for their company product, Weeve, detailed four production-hardening patterns for agentic pipelines after a load test hung for eleven minutes and then crashed. The patterns — exponential-backoff retries, fallback paths to cached data, loop detection over a sliding window of agent steps, and structured observability — address tool call failures, context overflow, silent loops, and hallucinated JSON that demos on clean inputs never surface.

Agent demos work. Production agents break — tool failures, context overflow, silent loops, hallucinated JSON. Four patterns that close the gap: retry, fallback, loop detection, and structured observability. The demo worked. Flawlessly. The agent found the customer record, pulled the transaction history, summarized it, and drafted the response in under four seconds. We showed it to the team on a Friday. Monday morning, in a production load test, it hung. For eleven minutes. Then crashed. Agent demos work because they run on clean inputs, stable APIs, and a single user. Production means flaky third-party tool endpoints, context windows that overflow mid-conversation, loops that spin silently because no exit condition matched, and hallucinations that look like valid JSON until you try to reserialize them. I ran into all four building the AI chatbot for my company product, Weeve. Here is what I added to close the gap. Tool call failures are the most common production failure mode. An external API returns a 503. A database query times out under load. The agent's default behaviour: propagate the error and crash the run. // Generic retry wrapper -- works around any async function, not just agent tool calls async function withRetry<T fn: = Promise<T , { maxAttempts = 3, baseDelayMs = 500 }: { maxAttempts?: number; baseDelayMs?: number } = {} : Promise<T { for let attempt = 1; attempt <= maxAttempts; attempt++ { try { return await fn } catch err { if attempt === maxAttempts throw err // Last attempt -- let it propagate // Math.pow 2, attempt - 1 : 1x, 2x, 4x -- backs off exponentially const delay = baseDelayMs Math.pow 2, attempt - 1 await new Promise res = setTimeout res, delay } } throw new Error 'unreachable' // TypeScript needs this -- the loop always returns or throws } // Wrap any tool call -- the agent code stays unchanged const result = await withRetry = externalCrmApi.getCustomer customerId , { maxAttempts: 3, baseDelayMs: 300 } Double the delay on each retry exponential backoff to avoid hammering an already-struggling API. For non-idempotent operations — writes, payments — don't retry automatically. Flag them for human review instead. Retries handle transient failures. Persistent failures need a fallback path — something the agent can do when the primary tool is unavailable, rather than returning nothing. js async function withFallback<T primary: = Promise<T , fallback: = Promise<T : Promise<T { try { // Try primary with a quick retry budget -- don't wait too long before giving up return await withRetry primary, { maxAttempts: 2, baseDelayMs: 200 } } catch { // Primary exhausted its retries -- fall back instead of propagating the error return await fallback } } // Real usage: live data - cached snapshot when live is unavailable const customerData = await withFallback = liveApi.getCustomer customerId , = cache.getLastKnownCustomer customerId // Stale but useful Fallbacks keep the agent useful under degraded conditions. A cached snapshot is worse than live data — but it is infinitely better than a crashed run and a blank response. Infinite loops are silent in agentic systems. The agent calls a tool, gets an ambiguous result, decides it needs more information, calls the same tool again. No exception is raised. The loop spins until the context window overflows or the timeout kills it. interface AgentStep { toolName: string input: string } // windowSize: only check the last N steps -- agents can revisit tools legitimately over a long run function detectLoop steps: AgentStep , windowSize = 6 : boolean { if steps.length < windowSize return false const recent = steps.slice -windowSize const seen = new Set<string for const step of recent { // Fingerprint = tool name + serialised input -- catches exact repeats const key = ${step.toolName}:${step.input} if seen.has key return true // Same call twice in the window = loop seen.add key } return false } // Check after every step in the agent run loop if detectLoop executedSteps { throw new Error Agent loop detected after ${executedSteps.length} steps -- breaking run } Fingerprint each step as tool name + serialised input. If you see the same fingerprint twice in a short window, you have a loop. Break it loudly — an error you can catch and log is better than a silent timeout. Logging 'agent failed' is not observability. Observability for an agent run means knowing: which tool was called, with what arguments, how long it took, what it returned, and what the agent decided to do next. interface AgentSpan { runId: string // Ties all spans for one agent run together step: number // Execution order within the run toolName: string inputSummary: string durationMs: number success: boolean errorMessage?: string outputSummary?: string } // Wrap every tool call with this -- don't call tools directly async function tracedToolCall runId: string, step: number, toolName: string, args: unknown, fn: = Promise<unknown : Promise<unknown { const start = Date.now const inputSummary = JSON.stringify args .slice 0, 200 // Truncate large inputs try { const result = await fn await observabilityStore.write { runId, step, toolName, inputSummary, durationMs: Date.now - start, success: true, outputSummary: JSON.stringify result .slice 0, 200 , } satisfies AgentSpan // satisfies: compiler checks shape without widening the type return result } catch err { await observabilityStore.write { runId, step, toolName, inputSummary, durationMs: Date.now - start, success: false, errorMessage: err instanceof Error ? err.message : String err , } satisfies AgentSpan throw err // Re-throw so the agent run still sees the failure } } Every tool call is a span. The runId ties all spans for a single agent run together — so when something fails, you can reconstruct exactly what happened, in order. This is the difference between debugging production and guessing about it. In production, these four patterns compose. The agent run starts with a runId. Every tool call goes through tracedToolCall, which wraps retry and fallback internally, logs each attempt as a span, and records failures without crashing the run. After each step, the loop detector scans the last six steps. If a loop is found, it throws — which gets caught, logged as a terminal span, and returned to the caller as a structured error rather than a silent hang. The result: agent runs that degrade gracefully, fail loudly, and leave a breadcrumb trail you can actually follow. Not the demo. The thing that runs on Monday morning. Run all four patterns yourself — retry, fallback, loop detection, and observability wired together