The caller gave up ten minutes ago: orphaned retries in agent fleets

A developer's blog post on Loop & Retry highlights a hidden failure mode in LLM agent fleets: orphaned retries that continue working after the caller has abandoned the request. The post proposes treating cancellation as an explicit input to retry loops, using cancellable events and backoff to stop wasted work. It notes that this approach works within a process but requires distributed tracing or similar mechanisms across network boundaries.

Originally published on Loop & Retry — field notes on building LLM agents that survive production. Bounding a fleet's retry spend https://loopandretry.github.io/posts/fleet-retry-patterns/?ref=devto assumes every retry loop is working toward something someone still wants. Shared budgets, circuit breakers, decorrelated jitter, dead-letter quarantine — all four patterns cap how much a fleet spends recovering from failure. None of them ask whether the work is still wanted at all. That's a different leak, and it doesn't show up in the same graphs: a retry that succeeds is not waste by any of those metrics, even when the caller stopped listening three retries ago. A user asks an agent a question, gets impatient, and closes the tab. Upstream, that agent had fanned out to four sub-agents, one of which was three retries into a flaky tool call. The tab closing never reaches that sub-agent. It has no way to know the parent is gone — it just sees a 503 , waits its backoff, and tries again. Eventually it succeeds, writes a result to a queue nobody drains, and exits looking healthy: no error, no timeout, no line in a failure dashboard. The cost was real and the outcome was invisible. This is different from the failures the fleet-retry patterns https://loopandretry.github.io/posts/fleet-retry-patterns/?ref=devto post targets. Those are about a dependency that's down — the fix is to stop calling it. This is about a dependency that's fine and a caller that's gone — the fix has to travel in the opposite direction, from parent to child, and most retry code has no channel for it. Looks fine. Retries a flaky call, bounded, with backoff. Has no idea if anyone still wants the answer. def fetch with retry client, request, max attempts=5 : delay = 0.5 for attempt in range max attempts : try: return client.call request except Transient: time.sleep delay delay = 2 raise RetriesExhausted Every one of those time.sleep calls is a window where a cancellation could have landed and didn't, because the function was never given anywhere to check for one. The fix is to treat "does anyone still want this" as an explicit input to the retry loop, checked on every iteration — not inferred from the absence of an exception. In Python, that's a token the caller can flip, threaded through every layer of the fan-out: python import asyncio async def fetch with retry client, request, cancel: asyncio.Event, max attempts=5 : delay = 0.5 for attempt in range max attempts : if cancel.is set : raise Cancelled f"abandoned after {attempt} attempts" try: return await client.call request except Transient: try: await asyncio.wait for cancel.wait , timeout=delay raise Cancelled f"abandoned during backoff, attempt {attempt}" except asyncio.TimeoutError: pass backoff elapsed, no cancellation — loop and retry delay = 2 raise RetriesExhausted Two things changed. First, the cancellation check happens before every attempt, not just at the top of the function — a long retry loop is a long-lived thing, and checking once at entry is checking once for a process that might run for minutes. Second, the backoff sleep itself is cancellable: asyncio.wait for cancel.wait , timeout=delay waits for either the backoff to elapse or a cancellation to arrive, whichever comes first, instead of blocking through it. A retry loop that only checks between attempts still burns the full backoff window on a request nobody wants. The event-token version works within one process. It falls apart the moment the fan-out crosses a boundary — a sub-agent invoked over HTTP, a task handed to a queue, a tool call routed through another service — because asyncio.Event doesn't survive a network hop. This is the same problem distributed tracing solved for observability, applied to control flow instead of just logging: the cancellation has to ride along as data, not as a language-level primitive. The deadline or a cancellation token travels IN the request, the same way a trace ID does — because it has to survive a network hop. async def call subagent client, request, deadline: float : remaining = deadline - time.monotonic if remaining <= 0: raise Cancelled "deadline already passed before dispatch" return await client.call request, headers={"X-Deadline": str deadline }, timeout=remaining, On the receiving side, the sub-agent's own retry loop reads the same deadline out of the request it was handed — not a fresh timeout of its own. async def handle request, headers : deadline = float headers "X-Deadline" return await fetch with retry until deadline, ... The header is the whole trick: a deadline or a cancellation token, if you need push rather than a fixed horizon is a value, so it composes across process boundaries the same way a trace ID or an idempotency key does. Without it, every hop in the fan-out invents its own timeout relative to when it started — which means a sub-agent five hops deep can easily outlive the root request by minutes, still retrying against a deadline that was never its own to set. A dropped HTTP retry after a client disconnect wastes a few hundred milliseconds of compute — a rounding error. An orphaned agent retry wastes a model call: the full-transcript re-read that makes long runs expensive https://loopandretry.github.io/posts/long-agent-runs-are-quadratic/?ref=devto doesn't get cheaper because nobody's going to read the output. Multiply by a fan-out of sub-agents, each with their own retry budget, and an abandoned request can keep spending for the exact backoff-and-retry duration you sized for legitimate transient failures — the retry budget math https://loopandretry.github.io/posts/retry-budgets/?ref=devto still applies, it's just amortized over zero delivered value instead of some. It also breaks the observability half of the story. Measuring failure modes in production https://loopandretry.github.io/posts/measuring-agent-failure-in-production/?ref=devto depends on every run ending in a labeled outcome — success, retryable failure, hard failure. An orphaned retry that eventually succeeds ends in a fifth, unlabeled bucket: nobody was there to receive it. That bucket doesn't fire alerts, doesn't increment an error counter, and doesn't show up in a retry-success metric, because by every metric those systems track, the call worked. It just worked for an audience of nobody, on a bill someone still pays. To catch this, you need idempotency keys and retry-attempt correlation https://loopandretry.github.io/posts/retry-context-observability/?ref=devto tracking whether a retry is part of a still-live request, not just whether it succeeded technically. A retry loop that never checks whether its caller is still around isn't robust, it's blind: cancellation has to be an explicit, checked-every-iteration input — not an inferred absence of exceptions — and it has to travel as data a deadline or token in the request across every process boundary the fan-out crosses, the same way a trace ID does. The question worth asking of any long-running fan-out: if the root caller vanished right now, how many hops deep would that news actually travel before something noticed? If the honest answer is "zero," you're not paying for resilience — you're paying to finish work for nobody.