Originally published on Loop & Retry — field notes on building LLM agents that survive production.
Bounding a fleet's retry spend assumes every retry loop is working toward something someone still wants. Shared budgets, circuit breakers, decorrelated jitter, dead-letter quarantine — all four patterns cap how much a fleet spends recovering from failure. None of them ask whether the work is still wanted at all. That's a different leak, and it doesn't show up in the same graphs: a retry that succeeds is not waste by any of those metrics, even when the caller stopped listening three retries ago.
A user asks an agent a question, gets impatient, and closes the tab. Upstream, that agent had fanned out to four sub-agents, one of which was three retries into a flaky tool call. The tab closing never reaches that sub-agent. It has no way to know the parent is gone — it just sees a 503
, waits its backoff, and tries again. Eventually it succeeds, writes a result to a queue nobody drains, and exits looking healthy: no error, no timeout, no line in a failure dashboard. The cost was real and the outcome was invisible.
This is different from the failures the fleet-retry patterns post targets. Those are about a dependency that's down — the fix is to stop calling it. This is about a dependency that's fine and a caller that's gone — the fix has to travel in the opposite direction, from parent to child, and most retry code has no channel for it.
def fetch_with_retry(client, request, max_attempts=5):
delay = 0.5
for attempt in range(max_attempts):
try:
return client.call(request)
except Transient:
time.sleep(delay)
delay *= 2
raise RetriesExhausted()
Every one of those time.sleep
calls is a window where a cancellation could have landed and didn't, because the function was never given anywhere to check for one.
The fix is to treat "does anyone still want this" as an explicit input to the retry loop, checked on every iteration — not inferred from the absence of an exception. In Python, that's a token the caller can flip, threaded through every layer of the fan-out:
import asyncio
async def fetch_with_retry(client, request, cancel: asyncio.Event, max_attempts=5):
delay = 0.5
for attempt in range(max_attempts):
if cancel.is_set():
raise Cancelled(f"abandoned after {attempt} attempts")
try:
return await client.call(request)
except Transient:
try:
await asyncio.wait_for(cancel.wait(), timeout=delay)
raise Cancelled(f"abandoned during backoff, attempt {attempt}")
except asyncio.TimeoutError:
pass # backoff elapsed, no cancellation — loop and retry
delay *= 2
raise RetriesExhausted()
Two things changed. First, the cancellation check happens before every attempt, not just at the top of the function — a long retry loop is a long-lived thing, and checking once at entry is checking once for a process that might run for minutes. Second, the backoff sleep itself is cancellable: asyncio.wait_for(cancel.wait(), timeout=delay)
waits for either the backoff to elapse or a cancellation to arrive, whichever comes first, instead of blocking through it. A retry loop that only checks between attempts still burns the full backoff window on a request nobody wants.
The event-token version works within one process. It falls apart the moment the fan-out crosses a boundary — a sub-agent invoked over HTTP, a task handed to a queue, a tool call routed through another service — because asyncio.Event
doesn't survive a network hop. This is the same problem distributed tracing solved for observability, applied to control flow instead of just logging: the cancellation has to ride along as data, not as a language-level primitive.
async def call_subagent(client, request, deadline: float):
remaining = deadline - time.monotonic()
if remaining <= 0:
raise Cancelled("deadline already passed before dispatch")
return await client.call(
request,
headers={"X-Deadline": str(deadline)},
timeout=remaining,
)
async def handle(request, headers):
deadline = float(headers["X-Deadline"])
return await fetch_with_retry_until(deadline, ...)
The header is the whole trick: a deadline (or a cancellation token, if you need push rather than a fixed horizon) is a value, so it composes across process boundaries the same way a trace ID or an idempotency key does. Without it, every hop in the fan-out invents its own timeout relative to when it started — which means a sub-agent five hops deep can easily outlive the root request by minutes, still retrying against a deadline that was never its own to set.
A dropped HTTP retry after a client disconnect wastes a few hundred milliseconds of compute — a rounding error. An orphaned agent retry wastes a model call: the full-transcript re-read that makes long runs expensive doesn't get cheaper because nobody's going to read the output. Multiply by a fan-out of sub-agents, each with their own retry budget, and an abandoned request can keep spending for the exact backoff-and-retry duration you sized for legitimate transient failures — the retry budget math still applies, it's just amortized over zero delivered value instead of some.
It also breaks the observability half of the story. Measuring failure modes in production depends on every run ending in a labeled outcome — success, retryable failure, hard failure. An orphaned retry that eventually succeeds ends in a fifth, unlabeled bucket: nobody was there to receive it. That bucket doesn't fire alerts, doesn't increment an error counter, and doesn't show up in a retry-success metric, because by every metric those systems track, the call worked. It just worked for an audience of nobody, on a bill someone still pays. To catch this, you need idempotency keys and retry-attempt correlation tracking whether a retry is part of a still-live request, not just whether it succeeded technically.
A retry loop that never checks whether its caller is still around isn't robust, it's blind: cancellation has to be an explicit, checked-every-iteration input — not an inferred absence of exceptions — and it has to travel as data (a deadline or token in the request) across every process boundary the fan-out crosses, the same way a trace ID does. The question worth asking of any long-running fan-out: if the root caller vanished right now, how many hops deep would that news actually travel before something noticed? If the honest answer is "zero," you're not paying for resilience — you're paying to finish work for nobody.