{"slug": "retry-context-building-observability-into-retry-decisions", "title": "Retry context: building observability into retry decisions", "summary": "A developer's blog post on Loop & Retry details how to build observability into retry decisions for LLM agents in production. The post emphasizes that retry attempts are indistinguishable between transient failures and permanent ones without visibility, and introduces idempotency keys and retry-attempt correlation as solutions. It provides code examples for stable idempotency key construction and logging attempt sequences to trace logical operations across retries.", "body_md": "*Originally published on Loop & Retry — field notes on building LLM agents that survive production.*\n\n[The fleet-patterns post](https://loopandretry.github.io/posts/fleet-retry-patterns/?ref=devto) explained the patterns that keep a fleet from drowning in its own retries. [The when-to-give-up post](https://loopandretry.github.io/posts/retry-patterns-when-to-give-up/?ref=devto) showed when to retry at all — which layer gets to make the decision, and why most code doesn't ask the right questions. (Both of those build on [the cost model](https://loopandretry.github.io/posts/retry-budgets/?ref=devto) and [a real incident](https://loopandretry.github.io/posts/postmortem-200-dollars-retrying-a-400/?ref=devto) that motivated it.) This post is about what you need to *know* to make that decision trustworthy in the first place.\n\nThe core problem: a retry attempt looks the same whether it's recovering from a transient network hiccup or whether you're trapped in a loop burning your [budget](https://loopandretry.github.io/posts/retry-budgets/?ref=devto) on a permanent failure. Without visibility into what's happening, your retry logic is guessing.\n\nAn idempotency key is a unique token attached to a request that tells the system \"if you've seen this before, return the cached result instead of replaying the work.\" It's not directly a retry concern — it's a consequence of a deeper rule: **retries are only safe when the operation is idempotent, and idempotency is only verifiable if the operation is labeled with a stable identity.**\n\n``` python\nimport uuid, time\n\nclass RetryableCall:\n    def __init__(self, operation_name, user_id, resource_id):\n        self.idempotency_key = f\"{user_id}:{resource_id}:{operation_name}:{int(time.time() * 1000)}\"\n        # ^ stable within a retry window (e.g., one second), fresh across retries >1s apart\n        self.attempt = 0\n\n    def call(self, client):\n        self.attempt += 1\n        headers = {\"X-Idempotency-Key\": self.idempotency_key}\n        return client.do_work(headers=headers)\n```\n\nThe key construction matters. If your idempotency key is just a UUID per *task* (not per *attempt*), all retries of the same task share it — which is what you want. If it's per-millisecond, two retries 100ms apart get different keys, and the server will double-process. The stability window should match your retry window: if you retry up to 5 seconds, the key should be stable for 5+ seconds.\n\nThis is how you tell the server \"this is attempt 3, but if you cached the result from attempt 1, use that.\" Without it, your retry is a replay: the server genuinely executes the operation twice.\n\nOnce you own idempotency, you need to track *which attempt* is which. This is where retry-attempt correlation comes in — a log trace that chains a series of attempts together so a human (or a monitoring system) can follow the story of a single logical operation through its retries.\n\n``` python\nclass CorrelatedRetry:\n    def __init__(self, logical_op_id):\n        self.logical_op_id = logical_op_id      # stable across all attempts of this op\n        self.attempt_sequence = []\n\n    def log_attempt(self, attempt_num, latency_ms, status, error=None):\n        entry = {\n            \"logical_op_id\": self.logical_op_id,\n            \"attempt\": attempt_num,\n            \"latency_ms\": latency_ms,\n            \"status\": status,\n            \"error\": error,\n            \"timestamp\": time.time(),\n        }\n        self.attempt_sequence.append(entry)\n        # emit to logs / traces\n        logger.info(\"retry_attempt\", extra=entry)\n\n# Example usage:\nlogical_id = str(uuid.uuid4())\nretry_tracer = CorrelatedRetry(logical_id)\n\nfor attempt in range(1, max_attempts + 1):\n    try:\n        start = time.monotonic()\n        result = call_downstream()\n        latency = (time.monotonic() - start) * 1000\n        retry_tracer.log_attempt(attempt, latency, \"success\")\n        return result\n    except Exception as e:\n        latency = (time.monotonic() - start) * 1000\n        retry_tracer.log_attempt(attempt, latency, \"failed\", str(e))\n        if attempt == max_attempts:\n            raise\n```\n\nWhat does this buy you? When a request fails after 3 retries, you can ask: \"Did each attempt fail for the same reason, or did they fail differently?\" If all three attempts get the same error (e.g., \"rate limit: retry after 60s\"), you know the failure is not transient — it's a real limit you've hit. If the first fails with a timeout and the second succeeds, you have evidence the transient was actually transient and the retry worked.\n\nThe logs become your debugging surface. When ops calls and says \"this user's transaction failed,\" you trace by `logical_op_id`\n\nand see not just \"failed: 500\" but \"attempt 1: timeout after 3.2s; attempt 2: timeout after 2.8s; attempt 3: failed upstream rate limit.\"\n\nThe retry budget bounds HOW MUCH you retry. Cost accounting tells you WHAT you're spending — and whether the budget is actually protecting you or you're gaming it.\n\n``` python\nclass RetryBudgetObserver:\n    def __init__(self, budget_name):\n        self.budget_name = budget_name\n        self.total_attempts = 0\n        self.successful_retries = 0        # retries that led to success\n        self.failed_retries = 0            # retries that failed\n        self.abandoned_retries = 0         # retries we didn't attempt (budget exhausted)\n        self.total_cost_usd = 0.0\n\n    def on_retry_attempt(self, cost_usd, eventual_success=None):\n        self.total_attempts += 1\n        self.total_cost_usd += cost_usd\n\n        if eventual_success is None:\n            self.abandoned_retries += 1  # budget said no\n        elif eventual_success:\n            self.successful_retries += 1\n        else:\n            self.failed_retries += 1\n\n    def report(self):\n        roi = (self.successful_retries / max(1, self.successful_retries + self.failed_retries)) if self.successful_retries + self.failed_retries > 0 else 0\n        return {\n            \"budget_name\": self.budget_name,\n            \"total_attempts\": self.total_attempts,\n            \"successful_retries\": self.successful_retries,\n            \"failed_retries\": self.failed_retries,\n            \"abandoned\": self.abandoned_retries,\n            \"roi\": roi,                       # success rate of attempts we made\n            \"cost_usd\": self.total_cost_usd,\n            \"cost_per_success\": self.total_cost_usd / max(1, self.successful_retries),\n        }\n```\n\nROI of 30% means one in three retry attempts recovered the operation. ROI of 5% means your budget is burning on dead ends — either your budget is too generous, or you're retrying things that won't ever succeed.\n\nThe cost-per-success metric is the one that matters most in production. If your cost per successful retry is $0.001 and your success rate is 95%, the budget is working. If it's $1.00 per success (because you're retrying expensive operations), the question becomes \"is that cost cheaper than the user's alternative?\" — a circuit breaker may make more sense than a retry budget at that scale.\n\nWhen all three pieces are wired up, the observability surface becomes active. You don't just log \"retry happened\" — you emit a structured record:\n\n```\n{\n  \"logical_op_id\": \"550e8400-e29b-41d4-a716-446655440000\",\n  \"operation\": \"process_transaction\",\n  \"idempotency_key\": \"user:12345:transaction:1721779200000\",\n  \"attempt_sequence\": [\n    {\n      \"attempt\": 1,\n      \"status\": \"timeout\",\n      \"latency_ms\": 30001,\n      \"downstream\": \"payment_svc\",\n      \"error\": \"read timeout after 30s\"\n    },\n    {\n      \"attempt\": 2,\n      \"status\": \"timeout\",\n      \"latency_ms\": 30002,\n      \"downstream\": \"payment_svc\",\n      \"error\": \"read timeout after 30s\"\n    },\n    {\n      \"attempt\": 3,\n      \"status\": \"rate_limit\",\n      \"latency_ms\": 245,\n      \"downstream\": \"payment_svc\",\n      \"error\": \"429: too many requests\"\n    }\n  ],\n  \"budget_name\": \"user_transactions\",\n  \"budget_decision\": \"abandon_further_retries\",\n  \"cost_usd\": 0.0023,\n  \"eventual_outcome\": \"failed\"\n}\n```\n\nThis record tells a story: \"The same operation hit two different failures — first timeouts (transient?), then a rate limit (hard limit). We stopped retrying. Cost: $0.0023, outcome: failed.\" A human reading this can decide: \"The rate limit kicked in after two timeouts — if we'd backed off longer, we might have succeeded. Or: the timeouts aren't transient, they're a symptom of cascade — backing off won't help.\"\n\nThe metrics that flow from this (success_rate per retry budget, cost_per_success, time_to_abandon) become the steering signal. When cost_per_success climbs above your threshold, it means your budget is chasing failures that won't resolve — time to tighten it. When success_rate drops, it means your transient assumptions are wrong — the errors you thought would pass aren't. [This is precisely what the measuring-success post](https://loopandretry.github.io/posts/retry-observability-measuring-success/?ref=devto) instruments: how to structure these metrics so they stay usable in production.\n\nThat's how you make a retry decision trustworthy: you instrument it so thoroughly that the logs themselves tell you whether the decision is working or not.", "url": "https://wpnews.pro/news/retry-context-building-observability-into-retry-decisions", "canonical_source": "https://dev.to/loopandretry/retry-context-building-observability-into-retry-decisions-4nf9", "published_at": "2026-08-16 22:02:25+00:00", "updated_at": "2026-08-16 22:42:14.346403+00:00", "lang": "en", "topics": ["large-language-models", "ai-agents", "developer-tools"], "entities": ["Loop & Retry"], "alternates": {"html": "https://wpnews.pro/news/retry-context-building-observability-into-retry-decisions", "markdown": "https://wpnews.pro/news/retry-context-building-observability-into-retry-decisions.md", "text": "https://wpnews.pro/news/retry-context-building-observability-into-retry-decisions.txt", "jsonld": "https://wpnews.pro/news/retry-context-building-observability-into-retry-decisions.jsonld"}}