Building reliable agentic pipelines — retries, fallbacks, and observability

Agent demos work. Production agents break — tool failures, context overflow, silent loops, hallucinated JSON. Four patterns that close the gap: retry, fallback, loop detection, and structured observability.

The demo worked. Flawlessly. The agent found the customer record, pulled the transaction history, summarized it, and drafted the response in under four seconds. We showed it to the team on a Friday. Monday morning, in a production load test, it hung. For eleven minutes. Then crashed.

Agent demos work because they run on clean inputs, stable APIs, and a single user. Production means flaky third-party tool endpoints, context windows that overflow mid-conversation, loops that spin silently because no exit condition matched, and hallucinations that look like valid JSON until you try to reserialize them.

I ran into all four building the AI chatbot for my company product, Weeve. Here is what I added to close the gap.

Tool call failures are the most common production failure mode. An external API returns a 503. A database query times out under load. The agent's default behaviour: propagate the error and crash the run.

// Generic retry wrapper -- works around any async function, not just agent tool calls
async function withRetry<T>(
  fn: () => Promise<T>,
  { maxAttempts = 3, baseDelayMs = 500 }: { maxAttempts?: number; baseDelayMs?: number } = {}
): Promise<T> {
  for (let attempt = 1; attempt <= maxAttempts; attempt++) {
    try {
      return await fn()
    } catch (err) {
      if (attempt === maxAttempts) throw err // Last attempt -- let it propagate
      // Math.pow(2, attempt - 1): 1x, 2x, 4x -- backs off exponentially
      const delay = baseDelayMs * Math.pow(2, attempt - 1)
      await new Promise(res => setTimeout(res, delay))
    }
  }
  throw new Error('unreachable') // TypeScript needs this -- the loop always returns or throws
}

// Wrap any tool call -- the agent code stays unchanged
const result = await withRetry(
  () => externalCrmApi.getCustomer(customerId),
  { maxAttempts: 3, baseDelayMs: 300 }
)

Double the delay on each retry (exponential backoff) to avoid hammering an already-struggling API. For non-idempotent operations — writes, payments — don't retry automatically. Flag them for human review instead.

Retries handle transient failures. Persistent failures need a fallback path — something the agent can do when the primary tool is unavailable, rather than returning nothing.

async function withFallback<T>(
  primary: () => Promise<T>,
  fallback: () => Promise<T>
): Promise<T> {
  try {
    // Try primary with a quick retry budget -- don't wait too long before giving up
    return await withRetry(primary, { maxAttempts: 2, baseDelayMs: 200 })
  } catch {
    // Primary exhausted its retries -- fall back instead of propagating the error
    return await fallback()
  }
}

// Real usage: live data -> cached snapshot when live is unavailable
const customerData = await withFallback(
  () => liveApi.getCustomer(customerId),
  () => cache.getLastKnownCustomer(customerId) // Stale but useful
)

Fallbacks keep the agent useful under degraded conditions. A cached snapshot is worse than live data — but it is infinitely better than a crashed run and a blank response.

Infinite loops are silent in agentic systems. The agent calls a tool, gets an ambiguous result, decides it needs more information, calls the same tool again. No exception is raised. The loop spins until the context window overflows or the timeout kills it.

interface AgentStep {
  toolName: string
  input: string
}

// windowSize: only check the last N steps -- agents can revisit tools legitimately over a long run
function detectLoop(steps: AgentStep[], windowSize = 6): boolean {
  if (steps.length < windowSize) return false
  const recent = steps.slice(-windowSize)
  const seen = new Set<string>()
  for (const step of recent) {
    // Fingerprint = tool name + serialised input -- catches exact repeats
    const key = `${step.toolName}:${step.input}`
    if (seen.has(key)) return true // Same call twice in the window = loop
    seen.add(key)
  }
  return false
}

// Check after every step in the agent run loop
if (detectLoop(executedSteps)) {
  throw new Error(
    `Agent loop detected after ${executedSteps.length} steps -- breaking run`
  )
}

Fingerprint each step as tool name + serialised input. If you see the same fingerprint twice in a short window, you have a loop. Break it loudly — an error you can catch and log is better than a silent timeout.

Logging 'agent failed' is not observability. Observability for an agent run means knowing: which tool was called, with what arguments, how long it took, what it returned, and what the agent decided to do next.

interface AgentSpan {
  runId: string     // Ties all spans for one agent run together
  step: number      // Execution order within the run
  toolName: string
  inputSummary: string
  durationMs: number
  success: boolean
  errorMessage?: string
  outputSummary?: string
}

// Wrap every tool call with this -- don't call tools directly
async function tracedToolCall(
  runId: string,
  step: number,
  toolName: string,
  args: unknown,
  fn: () => Promise<unknown>
): Promise<unknown> {
  const start = Date.now()
  const inputSummary = JSON.stringify(args).slice(0, 200) // Truncate large inputs
  try {
    const result = await fn()
    await observabilityStore.write({
      runId, step, toolName, inputSummary,
      durationMs: Date.now() - start,
      success: true,
      outputSummary: JSON.stringify(result).slice(0, 200),
    } satisfies AgentSpan) // satisfies: compiler checks shape without widening the type
    return result
  } catch (err) {
    await observabilityStore.write({
      runId, step, toolName, inputSummary,
      durationMs: Date.now() - start,
      success: false,
      errorMessage: err instanceof Error ? err.message : String(err),
    } satisfies AgentSpan)
    throw err // Re-throw so the agent run still sees the failure
  }
}

Every tool call is a span. The runId ties all spans for a single agent run together — so when something fails, you can reconstruct exactly what happened, in order. This is the difference between debugging production and guessing about it.

In production, these four patterns compose. The agent run starts with a runId. Every tool call goes through tracedToolCall, which wraps retry and fallback internally, logs each attempt as a span, and records failures without crashing the run. After each step, the loop detector scans the last six steps. If a loop is found, it throws — which gets caught, logged as a terminal span, and returned to the caller as a structured error rather than a silent hang.

The result: agent runs that degrade gracefully, fail loudly, and leave a breadcrumb trail you can actually follow. Not the demo. The thing that runs on Monday morning.

Run all four patterns yourself — retry, fallback, loop detection, and observability wired together