When Free Inference Should Not Own the Retry Loop

A retry loop that consults free inference before it fires again is not resilience. It is a second outage, delayed by a model call that can stall, drift, or disagree with itself. Status codes, idempotency keys, and a hard budget already decide whether work is safe to repeat. A model does not improve that decision when the model is the least reliable hop on the path.

This is not an argument against using a language model to study failures after the fact. It is a boundary. Live retries are a control-plane act: they duplicate side effects, burn downstream quotas, and stretch user-visible latency. Free inference is a best-effort text generator. Those two jobs do not share a service-level objective, and pretending they do is how a brief 503 becomes a thundering herd with a narrative attached.

Timeouts, 429s, and connection resets already carry well-understood codes. Mapping those codes through a completion endpoint adds tail latency, non-determinism, and a dependency that fails in the same class of ways as the original call. The loop then retries the work and retries the advisor. Two clocks start, and only one of them belongs to the customer request.

The analogy is a smoke alarm that phones a remote panel to ask whether the smoke is interesting. The building does not get safer while the panel thinks. It gets more smoke. Retry logic that waits on a completion is the same delay, except the panel can also change its mind between attempts.

Red flags show up as design choices, not as slogans. The first is a loop whose break condition is a sentence. If the path reads a model reply and then selects retry, give_up, or page_oncall, the classifier has become the circuit breaker. Model output is not breaker state. Breaker state is a counter, a clock, and a policy a human can read in a diff.

The second flag is a retry allowed to touch anything that is not idempotent. Charging a card, sending mail, rotating a secret, or applying a migration cannot wait on a paraphrase of the error body. Free inference will, on some days, describe a unique-constraint violation as a transient blip. That description is not a permission slip.

The third flag is coupling the retry budget to the inference budget. When the model is slow, the loop waits. When the model is rate-limited, the loop guesses. When a free endpoint swaps a backend, the loop's so-called intelligence silently changes shape. None of those events are properties of the original failure. They are properties of a volunteer dependency.

The fourth flag is using the model to invent severity. A 429 is a 429. A checksum mismatch is not probably fine if retried from another region. Teams that let free inference rewrite severity discover that pages arrive late, or not at all, because a completion once chose a calmer adjective.

Better alternatives are boring on purpose. Classify the error with the transport: HTTP status, gRPC code, SQLSTATE, syscall errno. Map that class onto a tiny policy object a test can exhaust. Cap attempts. Cap wall-clock time. Jitter the sleep. Honor Retry-After when the server sends it. Treat unknown codes as terminal unless a written policy says otherwise. Persist an idempotency key before the first attempt so a replay cannot double-apply.

The following example is a proposal, not a measured production benchmark. It keeps inference off the hot path. The loop never constructs a prompt.

from __future__ import annotations

from dataclasses import dataclass
from enum import Enum
import random
import time
from typing import Callable, TypeVar

T = TypeVar("T")

class ErrorClass(Enum):
    TRANSIENT = "transient"
    RATE_LIMIT = "rate_limit"
    CONFLICT = "conflict"
    TERMINAL = "terminal"

@dataclass(frozen=True)
class RetryPolicy:
    max_attempts: int = 4
    max_sleep_s: float = 8.0
    base_sleep_s: float = 0.25
    deadline_s: float = 12.0

@dataclass
class AttemptRecord:
    attempt: int
    error_class: str | None
    slept_s: float
    elapsed_s: float
    outcome: str

def classify_status(status: int) -> ErrorClass:
    if status in {408, 502, 503, 504}:
        return ErrorClass.TRANSIENT
    if status == 429:
        return ErrorClass.RATE_LIMIT
    if status in {409, 412}:
        return ErrorClass.CONFLICT
    return ErrorClass.TERMINAL

def next_sleep(policy: RetryPolicy, attempt: int, retry_after: float | None) -> float:
    if retry_after is not None:
        return min(retry_after, policy.max_sleep_s)
    exp = policy.base_sleep_s * (2 ** max(0, attempt - 1))
    jitter = random.random() * policy.base_sleep_s
    return min(exp + jitter, policy.max_sleep_s)

def run_with_budget(
    op: Callable[[], tuple[int, T]],
    policy: RetryPolicy,
    clock: Callable[[], float] = time.monotonic,
    sleeper: Callable[[float], None] = time.sleep,
) -> tuple[T, list[AttemptRecord]]:
    started = clock()
    records: list[AttemptRecord] = []
    last_error: str | None = None

    for attempt in range(1, policy.max_attempts + 1):
        elapsed = clock() - started
        if elapsed >= policy.deadline_s:
            records.append(AttemptRecord(attempt, last_error, 0.0, elapsed, "deadline"))
            raise TimeoutError("retry deadline exceeded")

        status, value = op()
        kind = classify_status(status)
        elapsed = clock() - started
        if status < 400:
            records.append(AttemptRecord(attempt, None, 0.0, elapsed, "ok"))
            return value, records
        if kind is ErrorClass.TERMINAL or kind is ErrorClass.CONFLICT:
            records.append(AttemptRecord(attempt, kind.value, 0.0, elapsed, "stop"))
            if kind is ErrorClass.CONFLICT:
                raise RuntimeError("conflict is not retryable without a new idempotency key")
            raise RuntimeError(f"terminal status {status}")

        sleep_for = next_sleep(policy, attempt, retry_after=None)
        remaining = policy.deadline_s - (clock() - started)
        sleep_for = max(0.0, min(sleep_for, remaining))
        records.append(AttemptRecord(attempt, kind.value, sleep_for, elapsed, "retry"))
        last_error = kind.value
        if attempt == policy.max_attempts:
            break
        sleeper(sleep_for)

    raise TimeoutError("retry attempts exhausted")

A test can freeze time and the dice. That is the point. Retry policy that depends on a model cannot freeze the model's vocabulary, and a suite that cannot freeze its oracle is not a suite.

from retry_policy import RetryPolicy, run_with_budget

def test_does_not_retry_conflict():
    calls = {"n": 0}

    def op():
        calls["n"] += 1
        return 409, None

    try:
        run_with_budget(op, RetryPolicy(), sleeper=lambda _s: None)
        raise AssertionError("expected conflict to stop the loop")
    except RuntimeError as exc:
        assert "conflict" in str(exc)
    assert calls["n"] == 1

def test_transient_respects_max_attempts():
    calls = {"n": 0}

    def op():
        calls["n"] += 1
        return 503, None

    sleeps: list[float] = []
    try:
        run_with_budget(
            op,
            RetryPolicy(max_attempts=3, deadline_s=60),
            sleeper=lambda s: sleeps.append(s),
        )
        raise AssertionError("expected exhaustion")
    except TimeoutError:
        pass
    assert calls["n"] == 3
    assert len(sleeps) == 2

Run the example locally with a clock the suite controls:

python -m pytest test_retry_policy.py -q

Offline analysis is a different job. After the loop stops, a team may dump the AttemptRecord list, the raw response body, and the idempotency key into JSONL and ask a model to summarize clusters for humans. That summary must not write back into the policy. It may open a ticket. It may suggest a new status mapping. A person still merges the mapping.

The snippet below is a sketch, not a networked client. It reads JSONL and prints a prompt. It does not call anything during the retry.

import json
import sys

def build_prompt(path: str) -> str:
    rows = [json.loads(line) for line in open(path, encoding="utf-8")]
    compact = [
        {
            "error_class": r.get("error_class"),
            "outcome": r.get("outcome"),
            "status_hint": r.get("status_hint"),
        }
        for r in rows[:200]
    ]
    return (
        "Cluster these retry records by error_class and outcome. "
        "Do not propose live retry rules. "
        "List candidate status mappings for a human to review.\n\n"
        + json.dumps(compact, indent=2)
    )

if __name__ == "__main__":
    sys.stdout.write(build_prompt(sys.argv[1]))

Disclosure: This article was prepared as part of MonkeyCode's product outreach. MonkeyCode is an open-source project that offers free model access and a free server option, which is a reasonable desk for iterating on that offline prompt and the JSONL shape. It is not a component of the live retry path. A free completion that is useful at a desk is still the wrong process to block a payment capture or a deploy rollback.

A compact field guide fits in a table because the rule is mechanical.

Exit criteria are equally concrete. Leave this design when the failure is not a transport class. If the only way to know whether a document is complete is to read its language, the problem is not a retry loop. It is a parser, a schema, or a human queue. Move that work off the request path, version the classifier, give it an SLO, and still refuse to let it replay side effects. Leave this design when a regulator requires a logged, deterministic reason for every replay; a free model cannot sign that reason. Leave it when p95 of the downstream already sits near the user timeout, because even a fast completion steals budget that jitter needs.

Limitations follow from the same boundary. Deterministic classification misses semantic failures: a 200 that contains an error string, a partial write that returned 201, a vendor that uses HTTP 500 for user-not-found. Those bugs need contract tests and vendor-specific adapters, not a model stuffed into except Exception. The example policy also assumes the caller can supply an idempotency key and that op() returns a single status-plus-value pair. Streaming RPCs, multi-step sagas, and exactly-once queues need their own ledgers.

Who should not take this as a blanket ban: researchers comparing retry policies in a simulator, and products whose whole job is semantic recovery on sandbox data with no customer side effects. They can put a model in the loop because a wrong retry there does not duplicate a charge. Production traffic is not that sandbox.

The practical workflow stays split. The live loop remains stupid, tested, and budgeted. The model reads a file after the fire is out. The retry budget still ships as code.