Retry context: building observability into retry decisions

A developer's blog post on Loop & Retry details how to build observability into retry decisions for LLM agents in production. The post emphasizes that retry attempts are indistinguishable between transient failures and permanent ones without visibility, and introduces idempotency keys and retry-attempt correlation as solutions. It provides code examples for stable idempotency key construction and logging attempt sequences to trace logical operations across retries.

Originally published on Loop & Retry — field notes on building LLM agents that survive production. The fleet-patterns post https://loopandretry.github.io/posts/fleet-retry-patterns/?ref=devto explained the patterns that keep a fleet from drowning in its own retries. The when-to-give-up post https://loopandretry.github.io/posts/retry-patterns-when-to-give-up/?ref=devto showed when to retry at all — which layer gets to make the decision, and why most code doesn't ask the right questions. Both of those build on the cost model https://loopandretry.github.io/posts/retry-budgets/?ref=devto and a real incident https://loopandretry.github.io/posts/postmortem-200-dollars-retrying-a-400/?ref=devto that motivated it. This post is about what you need to know to make that decision trustworthy in the first place. The core problem: a retry attempt looks the same whether it's recovering from a transient network hiccup or whether you're trapped in a loop burning your budget https://loopandretry.github.io/posts/retry-budgets/?ref=devto on a permanent failure. Without visibility into what's happening, your retry logic is guessing. An idempotency key is a unique token attached to a request that tells the system "if you've seen this before, return the cached result instead of replaying the work." It's not directly a retry concern — it's a consequence of a deeper rule: retries are only safe when the operation is idempotent, and idempotency is only verifiable if the operation is labeled with a stable identity. python import uuid, time class RetryableCall: def init self, operation name, user id, resource id : self.idempotency key = f"{user id}:{resource id}:{operation name}:{int time.time 1000 }" ^ stable within a retry window e.g., one second , fresh across retries 1s apart self.attempt = 0 def call self, client : self.attempt += 1 headers = {"X-Idempotency-Key": self.idempotency key} return client.do work headers=headers The key construction matters. If your idempotency key is just a UUID per task not per attempt , all retries of the same task share it — which is what you want. If it's per-millisecond, two retries 100ms apart get different keys, and the server will double-process. The stability window should match your retry window: if you retry up to 5 seconds, the key should be stable for 5+ seconds. This is how you tell the server "this is attempt 3, but if you cached the result from attempt 1, use that." Without it, your retry is a replay: the server genuinely executes the operation twice. Once you own idempotency, you need to track which attempt is which. This is where retry-attempt correlation comes in — a log trace that chains a series of attempts together so a human or a monitoring system can follow the story of a single logical operation through its retries. python class CorrelatedRetry: def init self, logical op id : self.logical op id = logical op id stable across all attempts of this op self.attempt sequence = def log attempt self, attempt num, latency ms, status, error=None : entry = { "logical op id": self.logical op id, "attempt": attempt num, "latency ms": latency ms, "status": status, "error": error, "timestamp": time.time , } self.attempt sequence.append entry emit to logs / traces logger.info "retry attempt", extra=entry Example usage: logical id = str uuid.uuid4 retry tracer = CorrelatedRetry logical id for attempt in range 1, max attempts + 1 : try: start = time.monotonic result = call downstream latency = time.monotonic - start 1000 retry tracer.log attempt attempt, latency, "success" return result except Exception as e: latency = time.monotonic - start 1000 retry tracer.log attempt attempt, latency, "failed", str e if attempt == max attempts: raise What does this buy you? When a request fails after 3 retries, you can ask: "Did each attempt fail for the same reason, or did they fail differently?" If all three attempts get the same error e.g., "rate limit: retry after 60s" , you know the failure is not transient — it's a real limit you've hit. If the first fails with a timeout and the second succeeds, you have evidence the transient was actually transient and the retry worked. The logs become your debugging surface. When ops calls and says "this user's transaction failed," you trace by logical op id and see not just "failed: 500" but "attempt 1: timeout after 3.2s; attempt 2: timeout after 2.8s; attempt 3: failed upstream rate limit." The retry budget bounds HOW MUCH you retry. Cost accounting tells you WHAT you're spending — and whether the budget is actually protecting you or you're gaming it. python class RetryBudgetObserver: def init self, budget name : self.budget name = budget name self.total attempts = 0 self.successful retries = 0 retries that led to success self.failed retries = 0 retries that failed self.abandoned retries = 0 retries we didn't attempt budget exhausted self.total cost usd = 0.0 def on retry attempt self, cost usd, eventual success=None : self.total attempts += 1 self.total cost usd += cost usd if eventual success is None: self.abandoned retries += 1 budget said no elif eventual success: self.successful retries += 1 else: self.failed retries += 1 def report self : roi = self.successful retries / max 1, self.successful retries + self.failed retries if self.successful retries + self.failed retries 0 else 0 return { "budget name": self.budget name, "total attempts": self.total attempts, "successful retries": self.successful retries, "failed retries": self.failed retries, "abandoned": self.abandoned retries, "roi": roi, success rate of attempts we made "cost usd": self.total cost usd, "cost per success": self.total cost usd / max 1, self.successful retries , } ROI of 30% means one in three retry attempts recovered the operation. ROI of 5% means your budget is burning on dead ends — either your budget is too generous, or you're retrying things that won't ever succeed. The cost-per-success metric is the one that matters most in production. If your cost per successful retry is $0.001 and your success rate is 95%, the budget is working. If it's $1.00 per success because you're retrying expensive operations , the question becomes "is that cost cheaper than the user's alternative?" — a circuit breaker may make more sense than a retry budget at that scale. When all three pieces are wired up, the observability surface becomes active. You don't just log "retry happened" — you emit a structured record: { "logical op id": "550e8400-e29b-41d4-a716-446655440000", "operation": "process transaction", "idempotency key": "user:12345:transaction:1721779200000", "attempt sequence": { "attempt": 1, "status": "timeout", "latency ms": 30001, "downstream": "payment svc", "error": "read timeout after 30s" }, { "attempt": 2, "status": "timeout", "latency ms": 30002, "downstream": "payment svc", "error": "read timeout after 30s" }, { "attempt": 3, "status": "rate limit", "latency ms": 245, "downstream": "payment svc", "error": "429: too many requests" } , "budget name": "user transactions", "budget decision": "abandon further retries", "cost usd": 0.0023, "eventual outcome": "failed" } This record tells a story: "The same operation hit two different failures — first timeouts transient? , then a rate limit hard limit . We stopped retrying. Cost: $0.0023, outcome: failed." A human reading this can decide: "The rate limit kicked in after two timeouts — if we'd backed off longer, we might have succeeded. Or: the timeouts aren't transient, they're a symptom of cascade — backing off won't help." The metrics that flow from this success rate per retry budget, cost per success, time to abandon become the steering signal. When cost per success climbs above your threshold, it means your budget is chasing failures that won't resolve — time to tighten it. When success rate drops, it means your transient assumptions are wrong — the errors you thought would pass aren't. This is precisely what the measuring-success post https://loopandretry.github.io/posts/retry-observability-measuring-success/?ref=devto instruments: how to structure these metrics so they stay usable in production. That's how you make a retry decision trustworthy: you instrument it so thoroughly that the logs themselves tell you whether the decision is working or not.