When Free Inference Should Not Own the Retry Loop

A developer argues that live retry loops should not consult free inference endpoints, contending that model calls add tail latency, non-determinism, and a dependency that fails in the same ways as the original request. The post outlines four design red flags — model-driven break conditions, retries touching non-idempotent operations, coupling retry budgets to inference budgets, and using models to invent severity — and proposes a deterministic policy based on transport-level error codes, capped attempts, jittered sleeps, and persisted idempotency keys.

A retry loop that consults free inference before it fires again is not resilience. It is a second outage, delayed by a model call that can stall, drift, or disagree with itself. Status codes, idempotency keys, and a hard budget already decide whether work is safe to repeat. A model does not improve that decision when the model is the least reliable hop on the path. This is not an argument against using a language model to study failures after the fact. It is a boundary. Live retries are a control-plane act: they duplicate side effects, burn downstream quotas, and stretch user-visible latency. Free inference is a best-effort text generator. Those two jobs do not share a service-level objective, and pretending they do is how a brief 503 becomes a thundering herd with a narrative attached. Timeouts, 429s, and connection resets already carry well-understood codes. Mapping those codes through a completion endpoint adds tail latency, non-determinism, and a dependency that fails in the same class of ways as the original call. The loop then retries the work and retries the advisor. Two clocks start, and only one of them belongs to the customer request. The analogy is a smoke alarm that phones a remote panel to ask whether the smoke is interesting. The building does not get safer while the panel thinks. It gets more smoke. Retry logic that waits on a completion is the same delay, except the panel can also change its mind between attempts. Red flags show up as design choices, not as slogans. The first is a loop whose break condition is a sentence. If the path reads a model reply and then selects retry , give up , or page oncall , the classifier has become the circuit breaker. Model output is not breaker state. Breaker state is a counter, a clock, and a policy a human can read in a diff. The second flag is a retry allowed to touch anything that is not idempotent. Charging a card, sending mail, rotating a secret, or applying a migration cannot wait on a paraphrase of the error body. Free inference will, on some days, describe a unique-constraint violation as a transient blip. That description is not a permission slip. The third flag is coupling the retry budget to the inference budget. When the model is slow, the loop waits. When the model is rate-limited, the loop guesses. When a free endpoint swaps a backend, the loop's so-called intelligence silently changes shape. None of those events are properties of the original failure. They are properties of a volunteer dependency. The fourth flag is using the model to invent severity. A 429 is a 429. A checksum mismatch is not probably fine if retried from another region. Teams that let free inference rewrite severity discover that pages arrive late, or not at all, because a completion once chose a calmer adjective. Better alternatives are boring on purpose. Classify the error with the transport: HTTP status, gRPC code, SQLSTATE, syscall errno. Map that class onto a tiny policy object a test can exhaust. Cap attempts. Cap wall-clock time. Jitter the sleep. Honor Retry-After when the server sends it. Treat unknown codes as terminal unless a written policy says otherwise. Persist an idempotency key before the first attempt so a replay cannot double-apply. The following example is a proposal, not a measured production benchmark. It keeps inference off the hot path. The loop never constructs a prompt. python retry policy.py from future import annotations from dataclasses import dataclass from enum import Enum import random import time from typing import Callable, TypeVar T = TypeVar "T" class ErrorClass Enum : TRANSIENT = "transient" RATE LIMIT = "rate limit" CONFLICT = "conflict" TERMINAL = "terminal" @dataclass frozen=True class RetryPolicy: max attempts: int = 4 max sleep s: float = 8.0 base sleep s: float = 0.25 deadline s: float = 12.0 @dataclass class AttemptRecord: attempt: int error class: str | None slept s: float elapsed s: float outcome: str def classify status status: int - ErrorClass: if status in {408, 502, 503, 504}: return ErrorClass.TRANSIENT if status == 429: return ErrorClass.RATE LIMIT if status in {409, 412}: return ErrorClass.CONFLICT return ErrorClass.TERMINAL def next sleep policy: RetryPolicy, attempt: int, retry after: float | None - float: if retry after is not None: return min retry after, policy.max sleep s exp = policy.base sleep s 2 max 0, attempt - 1 jitter = random.random policy.base sleep s return min exp + jitter, policy.max sleep s def run with budget op: Callable , tuple int, T , policy: RetryPolicy, clock: Callable , float = time.monotonic, sleeper: Callable float , None = time.sleep, - tuple T, list AttemptRecord : started = clock records: list AttemptRecord = last error: str | None = None for attempt in range 1, policy.max attempts + 1 : elapsed = clock - started if elapsed = policy.deadline s: records.append AttemptRecord attempt, last error, 0.0, elapsed, "deadline" raise TimeoutError "retry deadline exceeded" status, value = op kind = classify status status elapsed = clock - started if status < 400: records.append AttemptRecord attempt, None, 0.0, elapsed, "ok" return value, records if kind is ErrorClass.TERMINAL or kind is ErrorClass.CONFLICT: records.append AttemptRecord attempt, kind.value, 0.0, elapsed, "stop" if kind is ErrorClass.CONFLICT: raise RuntimeError "conflict is not retryable without a new idempotency key" raise RuntimeError f"terminal status {status}" sleep for = next sleep policy, attempt, retry after=None remaining = policy.deadline s - clock - started sleep for = max 0.0, min sleep for, remaining records.append AttemptRecord attempt, kind.value, sleep for, elapsed, "retry" last error = kind.value if attempt == policy.max attempts: break sleeper sleep for raise TimeoutError "retry attempts exhausted" A test can freeze time and the dice. That is the point. Retry policy that depends on a model cannot freeze the model's vocabulary, and a suite that cannot freeze its oracle is not a suite. python test retry policy.py from retry policy import RetryPolicy, run with budget def test does not retry conflict : calls = {"n": 0} def op : calls "n" += 1 return 409, None try: run with budget op, RetryPolicy , sleeper=lambda s: None raise AssertionError "expected conflict to stop the loop" except RuntimeError as exc: assert "conflict" in str exc assert calls "n" == 1 def test transient respects max attempts : calls = {"n": 0} def op : calls "n" += 1 return 503, None sleeps: list float = try: run with budget op, RetryPolicy max attempts=3, deadline s=60 , sleeper=lambda s: sleeps.append s , raise AssertionError "expected exhaustion" except TimeoutError: pass assert calls "n" == 3 assert len sleeps == 2 Run the example locally with a clock the suite controls: python -m pytest test retry policy.py -q Offline analysis is a different job. After the loop stops, a team may dump the AttemptRecord list, the raw response body, and the idempotency key into JSONL and ask a model to summarize clusters for humans. That summary must not write back into the policy. It may open a ticket. It may suggest a new status mapping. A person still merges the mapping. The snippet below is a sketch, not a networked client. It reads JSONL and prints a prompt. It does not call anything during the retry. php offline cluster.py import json import sys def build prompt path: str - str: rows = json.loads line for line in open path, encoding="utf-8" compact = { "error class": r.get "error class" , "outcome": r.get "outcome" , "status hint": r.get "status hint" , } for r in rows :200 return "Cluster these retry records by error class and outcome. " "Do not propose live retry rules. " "List candidate status mappings for a human to review.\n\n" + json.dumps compact, indent=2 if name == " main ": sys.stdout.write build prompt sys.argv 1 Disclosure: This article was prepared as part of MonkeyCode's product outreach. MonkeyCode is an open-source project that offers free model access and a free server option, which is a reasonable desk for iterating on that offline prompt and the JSONL shape. It is not a component of the live retry path. A free completion that is useful at a desk is still the wrong process to block a payment capture or a deploy rollback. A compact field guide fits in a table because the rule is mechanical. | Signal on the live path | Treat as | Do not treat as | |---|---|---| | 408 / 502 / 503 / 504 | transient, budgeted retry | a prompt about whether the API is sad | | 429 plus Retry-After | sleep bounded by policy | a model-chosen backoff story | | 409 / 412 | stop, new key required | try a slightly different payload | | 4xx validation | terminal | a rewrite of the request body | | timeout of the model hop | proof the hop should not be there | extra retries of the original work | Exit criteria are equally concrete. Leave this design when the failure is not a transport class. If the only way to know whether a document is complete is to read its language, the problem is not a retry loop. It is a parser, a schema, or a human queue. Move that work off the request path, version the classifier, give it an SLO, and still refuse to let it replay side effects. Leave this design when a regulator requires a logged, deterministic reason for every replay; a free model cannot sign that reason. Leave it when p95 of the downstream already sits near the user timeout, because even a fast completion steals budget that jitter needs. Limitations follow from the same boundary. Deterministic classification misses semantic failures: a 200 that contains an error string, a partial write that returned 201, a vendor that uses HTTP 500 for user-not-found. Those bugs need contract tests and vendor-specific adapters, not a model stuffed into except Exception . The example policy also assumes the caller can supply an idempotency key and that op returns a single status-plus-value pair. Streaming RPCs, multi-step sagas, and exactly-once queues need their own ledgers. Who should not take this as a blanket ban: researchers comparing retry policies in a simulator, and products whose whole job is semantic recovery on sandbox data with no customer side effects. They can put a model in the loop because a wrong retry there does not duplicate a charge. Production traffic is not that sandbox. The practical workflow stays split. The live loop remains stupid, tested, and budgeted. The model reads a file after the fire is out. The retry budget still ships as code.