# When Free Inference Should Not Own the Retry Loop > Source: > Published: 2026-09-13 20:45:25+00:00 A retry loop that consults free inference before it fires again is not resilience. It is a second outage, delayed by a model call that can stall, drift, or disagree with itself. Status codes, idempotency keys, and a hard budget already decide whether work is safe to repeat. A model does not improve that decision when the model is the least reliable hop on the path. This is not an argument against using a language model to study failures after the fact. It is a boundary. Live retries are a control-plane act: they duplicate side effects, burn downstream quotas, and stretch user-visible latency. Free inference is a best-effort text generator. Those two jobs do not share a service-level objective, and pretending they do is how a brief 503 becomes a thundering herd with a narrative attached. Timeouts, 429s, and connection resets already carry well-understood codes. Mapping those codes through a completion endpoint adds tail latency, non-determinism, and a dependency that fails in the same class of ways as the original call. The loop then retries the work *and* retries the advisor. Two clocks start, and only one of them belongs to the customer request. The analogy is a smoke alarm that phones a remote panel to ask whether the smoke is interesting. The building does not get safer while the panel thinks. It gets more smoke. Retry logic that waits on a completion is the same delay, except the panel can also change its mind between attempts. Red flags show up as design choices, not as slogans. The first is a loop whose break condition is a sentence. If the path reads a model reply and then selects `retry`, `give_up`, or `page_oncall`, the classifier has become the circuit breaker. Model output is not breaker state. Breaker state is a counter, a clock, and a policy a human can read in a diff. The second flag is a retry allowed to touch anything that is not idempotent. Charging a card, sending mail, rotating a secret, or applying a migration cannot wait on a paraphrase of the error body. Free inference will, on some days, describe a unique-constraint violation as a transient blip. That description is not a permission slip. The third flag is coupling the retry budget to the inference budget. When the model is slow, the loop waits. When the model is rate-limited, the loop guesses. When a free endpoint swaps a backend, the loop's so-called intelligence silently changes shape. None of those events are properties of the original failure. They are properties of a volunteer dependency. The fourth flag is using the model to invent severity. A 429 is a 429. A checksum mismatch is not probably fine if retried from another region. Teams that let free inference rewrite severity discover that pages arrive late, or not at all, because a completion once chose a calmer adjective. Better alternatives are boring on purpose. Classify the error with the transport: HTTP status, gRPC code, SQLSTATE, syscall errno. Map that class onto a tiny policy object a test can exhaust. Cap attempts. Cap wall-clock time. Jitter the sleep. Honor `Retry-After` when the server sends it. Treat unknown codes as terminal unless a written policy says otherwise. Persist an idempotency key before the first attempt so a replay cannot double-apply. The following example is a proposal, not a measured production benchmark. It keeps inference off the hot path. The loop never constructs a prompt. ``` python # retry_policy.py from __future__ import annotations from dataclasses import dataclass from enum import Enum import random import time from typing import Callable, TypeVar T = TypeVar("T") class ErrorClass(Enum): TRANSIENT = "transient" RATE_LIMIT = "rate_limit" CONFLICT = "conflict" TERMINAL = "terminal" @dataclass(frozen=True) class RetryPolicy: max_attempts: int = 4 max_sleep_s: float = 8.0 base_sleep_s: float = 0.25 deadline_s: float = 12.0 @dataclass class AttemptRecord: attempt: int error_class: str | None slept_s: float elapsed_s: float outcome: str def classify_status(status: int) -> ErrorClass: if status in {408, 502, 503, 504}: return ErrorClass.TRANSIENT if status == 429: return ErrorClass.RATE_LIMIT if status in {409, 412}: return ErrorClass.CONFLICT return ErrorClass.TERMINAL def next_sleep(policy: RetryPolicy, attempt: int, retry_after: float | None) -> float: if retry_after is not None: return min(retry_after, policy.max_sleep_s) exp = policy.base_sleep_s * (2 ** max(0, attempt - 1)) jitter = random.random() * policy.base_sleep_s return min(exp + jitter, policy.max_sleep_s) def run_with_budget( op: Callable[[], tuple[int, T]], policy: RetryPolicy, clock: Callable[[], float] = time.monotonic, sleeper: Callable[[float], None] = time.sleep, ) -> tuple[T, list[AttemptRecord]]: started = clock() records: list[AttemptRecord] = [] last_error: str | None = None for attempt in range(1, policy.max_attempts + 1): elapsed = clock() - started if elapsed >= policy.deadline_s: records.append(AttemptRecord(attempt, last_error, 0.0, elapsed, "deadline")) raise TimeoutError("retry deadline exceeded") status, value = op() kind = classify_status(status) elapsed = clock() - started if status < 400: records.append(AttemptRecord(attempt, None, 0.0, elapsed, "ok")) return value, records if kind is ErrorClass.TERMINAL or kind is ErrorClass.CONFLICT: records.append(AttemptRecord(attempt, kind.value, 0.0, elapsed, "stop")) if kind is ErrorClass.CONFLICT: raise RuntimeError("conflict is not retryable without a new idempotency key") raise RuntimeError(f"terminal status {status}") sleep_for = next_sleep(policy, attempt, retry_after=None) remaining = policy.deadline_s - (clock() - started) sleep_for = max(0.0, min(sleep_for, remaining)) records.append(AttemptRecord(attempt, kind.value, sleep_for, elapsed, "retry")) last_error = kind.value if attempt == policy.max_attempts: break sleeper(sleep_for) raise TimeoutError("retry attempts exhausted") ``` A test can freeze time and the dice. That is the point. Retry policy that depends on a model cannot freeze the model's vocabulary, and a suite that cannot freeze its oracle is not a suite. ``` python # test_retry_policy.py from retry_policy import RetryPolicy, run_with_budget def test_does_not_retry_conflict(): calls = {"n": 0} def op(): calls["n"] += 1 return 409, None try: run_with_budget(op, RetryPolicy(), sleeper=lambda _s: None) raise AssertionError("expected conflict to stop the loop") except RuntimeError as exc: assert "conflict" in str(exc) assert calls["n"] == 1 def test_transient_respects_max_attempts(): calls = {"n": 0} def op(): calls["n"] += 1 return 503, None sleeps: list[float] = [] try: run_with_budget( op, RetryPolicy(max_attempts=3, deadline_s=60), sleeper=lambda s: sleeps.append(s), ) raise AssertionError("expected exhaustion") except TimeoutError: pass assert calls["n"] == 3 assert len(sleeps) == 2 ``` Run the example locally with a clock the suite controls: ``` python -m pytest test_retry_policy.py -q ``` Offline analysis is a different job. After the loop stops, a team may dump the `AttemptRecord` list, the raw response body, and the idempotency key into JSONL and ask a model to summarize clusters for humans. That summary must not write back into the policy. It may open a ticket. It may suggest a new status mapping. A person still merges the mapping. The snippet below is a sketch, not a networked client. It reads JSONL and prints a prompt. It does not call anything during the retry. ``` php # offline_cluster.py import json import sys def build_prompt(path: str) -> str: rows = [json.loads(line) for line in open(path, encoding="utf-8")] compact = [ { "error_class": r.get("error_class"), "outcome": r.get("outcome"), "status_hint": r.get("status_hint"), } for r in rows[:200] ] return ( "Cluster these retry records by error_class and outcome. " "Do not propose live retry rules. " "List candidate status mappings for a human to review.\n\n" + json.dumps(compact, indent=2) ) if __name__ == "__main__": sys.stdout.write(build_prompt(sys.argv[1])) ``` Disclosure: This article was prepared as part of MonkeyCode's product outreach. MonkeyCode is an open-source project that offers free model access and a free server option, which is a reasonable desk for iterating on that offline prompt and the JSONL shape. It is not a component of the live retry path. A free completion that is useful at a desk is still the wrong process to block a payment capture or a deploy rollback. A compact field guide fits in a table because the rule is mechanical. | Signal on the live path | Treat as | Do not treat as | |---|---|---| | 408 / 502 / 503 / 504 | transient, budgeted retry | a prompt about whether the API is sad | | 429 plus Retry-After | sleep bounded by policy | a model-chosen backoff story | | 409 / 412 | stop, new key required | try a slightly different payload | | 4xx validation | terminal | a rewrite of the request body | | timeout of the model hop | proof the hop should not be there | extra retries of the original work | Exit criteria are equally concrete. Leave this design when the failure is not a transport class. If the only way to know whether a document is complete is to read its language, the problem is not a retry loop. It is a parser, a schema, or a human queue. Move that work off the request path, version the classifier, give it an SLO, and still refuse to let it replay side effects. Leave this design when a regulator requires a logged, deterministic reason for every replay; a free model cannot sign that reason. Leave it when p95 of the downstream already sits near the user timeout, because even a fast completion steals budget that jitter needs. Limitations follow from the same boundary. Deterministic classification misses semantic failures: a 200 that contains an error string, a partial write that returned 201, a vendor that uses HTTP 500 for user-not-found. Those bugs need contract tests and vendor-specific adapters, not a model stuffed into `except Exception`. The example policy also assumes the caller can supply an idempotency key and that `op()` returns a single status-plus-value pair. Streaming RPCs, multi-step sagas, and exactly-once queues need their own ledgers. Who should not take this as a blanket ban: researchers comparing retry policies in a simulator, and products whose whole job is semantic recovery on sandbox data with no customer side effects. They can put a model in the loop because a wrong retry there does not duplicate a charge. Production traffic is not that sandbox. The practical workflow stays split. The live loop remains stupid, tested, and budgeted. The model reads a file after the fire is out. The retry budget still ships as code.