{"slug": "when-free-inference-should-not-own-the-retry-loop", "title": "When Free Inference Should Not Own the Retry Loop", "summary": "A developer argues that live retry loops should not consult free inference endpoints, contending that model calls add tail latency, non-determinism, and a dependency that fails in the same ways as the original request. The post outlines four design red flags — model-driven break conditions, retries touching non-idempotent operations, coupling retry budgets to inference budgets, and using models to invent severity — and proposes a deterministic policy based on transport-level error codes, capped attempts, jittered sleeps, and persisted idempotency keys.", "body_md": "A retry loop that consults free inference before it fires again is not resilience. It is a second outage, delayed by a model call that can stall, drift, or disagree with itself. Status codes, idempotency keys, and a hard budget already decide whether work is safe to repeat. A model does not improve that decision when the model is the least reliable hop on the path.\n\nThis is not an argument against using a language model to study failures after the fact. It is a boundary. Live retries are a control-plane act: they duplicate side effects, burn downstream quotas, and stretch user-visible latency. Free inference is a best-effort text generator. Those two jobs do not share a service-level objective, and pretending they do is how a brief 503 becomes a thundering herd with a narrative attached.\n\nTimeouts, 429s, and connection resets already carry well-understood codes. Mapping those codes through a completion endpoint adds tail latency, non-determinism, and a dependency that fails in the same class of ways as the original call. The loop then retries the work *and* retries the advisor. Two clocks start, and only one of them belongs to the customer request.\n\nThe analogy is a smoke alarm that phones a remote panel to ask whether the smoke is interesting. The building does not get safer while the panel thinks. It gets more smoke. Retry logic that waits on a completion is the same delay, except the panel can also change its mind between attempts.\n\nRed flags show up as design choices, not as slogans. The first is a loop whose break condition is a sentence. If the path reads a model reply and then selects `retry`, `give_up`, or `page_oncall`, the classifier has become the circuit breaker. Model output is not breaker state. Breaker state is a counter, a clock, and a policy a human can read in a diff.\n\nThe second flag is a retry allowed to touch anything that is not idempotent. Charging a card, sending mail, rotating a secret, or applying a migration cannot wait on a paraphrase of the error body. Free inference will, on some days, describe a unique-constraint violation as a transient blip. That description is not a permission slip.\n\nThe third flag is coupling the retry budget to the inference budget. When the model is slow, the loop waits. When the model is rate-limited, the loop guesses. When a free endpoint swaps a backend, the loop's so-called intelligence silently changes shape. None of those events are properties of the original failure. They are properties of a volunteer dependency.\n\nThe fourth flag is using the model to invent severity. A 429 is a 429. A checksum mismatch is not probably fine if retried from another region. Teams that let free inference rewrite severity discover that pages arrive late, or not at all, because a completion once chose a calmer adjective.\n\nBetter alternatives are boring on purpose. Classify the error with the transport: HTTP status, gRPC code, SQLSTATE, syscall errno. Map that class onto a tiny policy object a test can exhaust. Cap attempts. Cap wall-clock time. Jitter the sleep. Honor `Retry-After` when the server sends it. Treat unknown codes as terminal unless a written policy says otherwise. Persist an idempotency key before the first attempt so a replay cannot double-apply.\n\nThe following example is a proposal, not a measured production benchmark. It keeps inference off the hot path. The loop never constructs a prompt.\n\n``` python\n# retry_policy.py\nfrom __future__ import annotations\n\nfrom dataclasses import dataclass\nfrom enum import Enum\nimport random\nimport time\nfrom typing import Callable, TypeVar\n\nT = TypeVar(\"T\")\n\nclass ErrorClass(Enum):\n    TRANSIENT = \"transient\"\n    RATE_LIMIT = \"rate_limit\"\n    CONFLICT = \"conflict\"\n    TERMINAL = \"terminal\"\n\n@dataclass(frozen=True)\nclass RetryPolicy:\n    max_attempts: int = 4\n    max_sleep_s: float = 8.0\n    base_sleep_s: float = 0.25\n    deadline_s: float = 12.0\n\n@dataclass\nclass AttemptRecord:\n    attempt: int\n    error_class: str | None\n    slept_s: float\n    elapsed_s: float\n    outcome: str\n\ndef classify_status(status: int) -> ErrorClass:\n    if status in {408, 502, 503, 504}:\n        return ErrorClass.TRANSIENT\n    if status == 429:\n        return ErrorClass.RATE_LIMIT\n    if status in {409, 412}:\n        return ErrorClass.CONFLICT\n    return ErrorClass.TERMINAL\n\ndef next_sleep(policy: RetryPolicy, attempt: int, retry_after: float | None) -> float:\n    if retry_after is not None:\n        return min(retry_after, policy.max_sleep_s)\n    exp = policy.base_sleep_s * (2 ** max(0, attempt - 1))\n    jitter = random.random() * policy.base_sleep_s\n    return min(exp + jitter, policy.max_sleep_s)\n\ndef run_with_budget(\n    op: Callable[[], tuple[int, T]],\n    policy: RetryPolicy,\n    clock: Callable[[], float] = time.monotonic,\n    sleeper: Callable[[float], None] = time.sleep,\n) -> tuple[T, list[AttemptRecord]]:\n    started = clock()\n    records: list[AttemptRecord] = []\n    last_error: str | None = None\n\n    for attempt in range(1, policy.max_attempts + 1):\n        elapsed = clock() - started\n        if elapsed >= policy.deadline_s:\n            records.append(AttemptRecord(attempt, last_error, 0.0, elapsed, \"deadline\"))\n            raise TimeoutError(\"retry deadline exceeded\")\n\n        status, value = op()\n        kind = classify_status(status)\n        elapsed = clock() - started\n        if status < 400:\n            records.append(AttemptRecord(attempt, None, 0.0, elapsed, \"ok\"))\n            return value, records\n        if kind is ErrorClass.TERMINAL or kind is ErrorClass.CONFLICT:\n            records.append(AttemptRecord(attempt, kind.value, 0.0, elapsed, \"stop\"))\n            if kind is ErrorClass.CONFLICT:\n                raise RuntimeError(\"conflict is not retryable without a new idempotency key\")\n            raise RuntimeError(f\"terminal status {status}\")\n\n        sleep_for = next_sleep(policy, attempt, retry_after=None)\n        remaining = policy.deadline_s - (clock() - started)\n        sleep_for = max(0.0, min(sleep_for, remaining))\n        records.append(AttemptRecord(attempt, kind.value, sleep_for, elapsed, \"retry\"))\n        last_error = kind.value\n        if attempt == policy.max_attempts:\n            break\n        sleeper(sleep_for)\n\n    raise TimeoutError(\"retry attempts exhausted\")\n```\n\nA test can freeze time and the dice. That is the point. Retry policy that depends on a model cannot freeze the model's vocabulary, and a suite that cannot freeze its oracle is not a suite.\n\n``` python\n# test_retry_policy.py\nfrom retry_policy import RetryPolicy, run_with_budget\n\ndef test_does_not_retry_conflict():\n    calls = {\"n\": 0}\n\n    def op():\n        calls[\"n\"] += 1\n        return 409, None\n\n    try:\n        run_with_budget(op, RetryPolicy(), sleeper=lambda _s: None)\n        raise AssertionError(\"expected conflict to stop the loop\")\n    except RuntimeError as exc:\n        assert \"conflict\" in str(exc)\n    assert calls[\"n\"] == 1\n\ndef test_transient_respects_max_attempts():\n    calls = {\"n\": 0}\n\n    def op():\n        calls[\"n\"] += 1\n        return 503, None\n\n    sleeps: list[float] = []\n    try:\n        run_with_budget(\n            op,\n            RetryPolicy(max_attempts=3, deadline_s=60),\n            sleeper=lambda s: sleeps.append(s),\n        )\n        raise AssertionError(\"expected exhaustion\")\n    except TimeoutError:\n        pass\n    assert calls[\"n\"] == 3\n    assert len(sleeps) == 2\n```\n\nRun the example locally with a clock the suite controls:\n\n```\npython -m pytest test_retry_policy.py -q\n```\n\nOffline analysis is a different job. After the loop stops, a team may dump the `AttemptRecord` list, the raw response body, and the idempotency key into JSONL and ask a model to summarize clusters for humans. That summary must not write back into the policy. It may open a ticket. It may suggest a new status mapping. A person still merges the mapping.\n\nThe snippet below is a sketch, not a networked client. It reads JSONL and prints a prompt. It does not call anything during the retry.\n\n``` php\n# offline_cluster.py\nimport json\nimport sys\n\ndef build_prompt(path: str) -> str:\n    rows = [json.loads(line) for line in open(path, encoding=\"utf-8\")]\n    compact = [\n        {\n            \"error_class\": r.get(\"error_class\"),\n            \"outcome\": r.get(\"outcome\"),\n            \"status_hint\": r.get(\"status_hint\"),\n        }\n        for r in rows[:200]\n    ]\n    return (\n        \"Cluster these retry records by error_class and outcome. \"\n        \"Do not propose live retry rules. \"\n        \"List candidate status mappings for a human to review.\\n\\n\"\n        + json.dumps(compact, indent=2)\n    )\n\nif __name__ == \"__main__\":\n    sys.stdout.write(build_prompt(sys.argv[1]))\n```\n\nDisclosure: This article was prepared as part of MonkeyCode's product outreach. MonkeyCode is an open-source project that offers free model access and a free server option, which is a reasonable desk for iterating on that offline prompt and the JSONL shape. It is not a component of the live retry path. A free completion that is useful at a desk is still the wrong process to block a payment capture or a deploy rollback.\n\nA compact field guide fits in a table because the rule is mechanical.\n\n| Signal on the live path | Treat as | Do not treat as | \n|---|---|---|\n| 408 / 502 / 503 / 504 | transient, budgeted retry | a prompt about whether the API is sad | \n| 429 plus Retry-After | sleep bounded by policy | a model-chosen backoff story | \n| 409 / 412 | stop, new key required | try a slightly different payload | \n| 4xx validation | terminal | a rewrite of the request body | \n| timeout of the model hop | proof the hop should not be there | extra retries of the original work | \n\nExit criteria are equally concrete. Leave this design when the failure is not a transport class. If the only way to know whether a document is complete is to read its language, the problem is not a retry loop. It is a parser, a schema, or a human queue. Move that work off the request path, version the classifier, give it an SLO, and still refuse to let it replay side effects. Leave this design when a regulator requires a logged, deterministic reason for every replay; a free model cannot sign that reason. Leave it when p95 of the downstream already sits near the user timeout, because even a fast completion steals budget that jitter needs.\n\nLimitations follow from the same boundary. Deterministic classification misses semantic failures: a 200 that contains an error string, a partial write that returned 201, a vendor that uses HTTP 500 for user-not-found. Those bugs need contract tests and vendor-specific adapters, not a model stuffed into `except Exception`. The example policy also assumes the caller can supply an idempotency key and that `op()` returns a single status-plus-value pair. Streaming RPCs, multi-step sagas, and exactly-once queues need their own ledgers.\n\nWho should not take this as a blanket ban: researchers comparing retry policies in a simulator, and products whose whole job is semantic recovery on sandbox data with no customer side effects. They can put a model in the loop because a wrong retry there does not duplicate a charge. Production traffic is not that sandbox.\n\nThe practical workflow stays split. The live loop remains stupid, tested, and budgeted. The model reads a file after the fire is out. The retry budget still ships as code.", "url": "https://wpnews.pro/news/when-free-inference-should-not-own-the-retry-loop", "canonical_source": "https://dev.to/aiio_6471/when-free-inference-should-not-own-the-retry-loop-2in3", "published_at": "2026-09-13 20:45:25+00:00", "updated_at": "2026-09-13 21:21:55.685951+00:00", "lang": "en", "topics": ["ai-infrastructure", "mlops", "ai-agents", "developer-tools"], "entities": [], "alternates": {"html": "https://wpnews.pro/news/when-free-inference-should-not-own-the-retry-loop", "markdown": "https://wpnews.pro/news/when-free-inference-should-not-own-the-retry-loop.md", "text": "https://wpnews.pro/news/when-free-inference-should-not-own-the-retry-loop.txt", "jsonld": "https://wpnews.pro/news/when-free-inference-should-not-own-the-retry-loop.jsonld"}}