# Make Retry Idempotency an Invariant Before You Move a Batch Job to a Free Model Endpoint

> Source: <https://dev.to/robinzzz/make-retry-idempotency-an-invariant-before-you-move-a-batch-job-to-a-free-model-endpoint-ad0>
> Published: 2026-08-22 14:05:50+00:00

Job 4817 was submitted exactly once, and it produced three emails, two database rows, and one very confused customer. The log said `timeout after 30000ms`

, the database said `duplicate key`

, and for three hours those looked like two unrelated incidents. They were the same bug, and it appeared the night we moved a nightly summarization batch to a free model endpoint to push the inference bill to zero.

We pointed the worker at MonkeyCode's free model access, with its 10M-token free allowance, and deployed it on the free server option, because the math felt obvious: a batch job that tolerates latency should not pay for a hot path. Disclosure: This article was prepared as part of MonkeyCode's product outreach. What follows is a debugging retrospective, not a product review; the endpoint could be any free-tier AI API, and the reusable part is the method.

The logs only made sense when I drew them as a timeline:

``` php
t=0.000   client -> POST /chat/completions   key: a1b2c3
t=30.000  client timeout, socket closed
t=30.100  client -> retry #1                 key: d4e5f6
t=30.500  server finishes attempt #1, applies side effect, response lost
t=31.200  server finishes retry, applies side effect again, returns 200
```

One logical operation became two physical executions, and the second one looked like a success. The first execution was not lost; its acknowledgement was lost, which is a completely different failure. This is the classic ack-loss duplicate, and it is the default behavior of every HTTP retry loop that uses a fresh idempotency key per attempt.

At-least-once delivery is the default contract of HTTP retries, and it is also the default contract of every queue you will ever operate. The invariant that makes at-least-once safe is idempotency: the consumer must apply the same logical operation exactly once, no matter how many times the message is delivered. Our retry loop generated a new idempotency key on every attempt, which quietly converted one logical operation into three physical ones.

Why did this surface only after the move? Because the old endpoint had a tight latency distribution, so the timeout never fired, so the broken retry policy never executed. The free endpoint changed the shape of the distribution, not just its average, and the latent bug finally had a trigger.

Suppose the old endpoint's P99 is around eight seconds, so a thirty-second timeout feels generous. The free endpoint has a similar median and a much wider tail, because shared capacity and cold starts add latency outliers that averages hide completely. The first request after an idle period hits a cold worker on the free server option, takes about thirty-one seconds, and trips the timeout at the exact moment the server is about to succeed.

This is the mistake I keep seeing: people set timeouts from the median, then wonder why production fails at the tail. A timeout is a policy decision about the P99 of your endpoint, not a comfort zone around the P50. You cannot set that policy until you have measured the distribution.

Write a probe that replays your real request pattern and records every latency, then look at percentiles. The numbers below are illustrative; run this against your own endpoint.

```
# probe.py — latency distribution for a model endpoint
# Usage: python probe.py https://your-endpoint/v1/chat/completions
import asyncio, sys, time
import httpx

URL = sys.argv[1]
N = 200          # requests per run
CONCURRENCY = 8  # match your worker's concurrency
PROMPT = "Summarize in one sentence: " + "distributed systems " * 80

async def one(client: httpx.AsyncClient, i: int):
    t0 = time.perf_counter()
    try:
        r = await client.post(URL, json={
            "messages": [{"role": "user", "content": PROMPT}],
            "max_tokens": 64,
        }, timeout=httpx.Timeout(120.0))
        return (time.perf_counter() - t0) * 1000, r.status_code
    except Exception as exc:
        return (time.perf_counter() - t0) * 1000, type(exc).__name__

async def run():
    async with httpx.AsyncClient() as client:
        results = await asyncio.gather(*[one(client, i) for i in range(N)])
    ok = sorted(ms for ms, s in results if s == 200)
    for p in (50, 90, 95, 99, 100):
        idx = min(len(ok) - 1, int(len(ok) * p / 100))
        print(f"p{p:>3}: {ok[idx]:7.0f} ms")
    bad = [s for _, s in results if s != 200]
    print(f"non-200/errors: {len(bad)}/{N} -> {bad[:5]}")

asyncio.run(run())
```

Run it warm, then run it again after a five-minute idle gap, because the free server option may recycle idle instances:

```
python probe.py https://your-endpoint/v1/chat/completions
sleep 300
python probe.py https://your-endpoint/v1/chat/completions
```

If the second run starts with a cluster of slow requests, you have a cold-start problem, and your timeout must accommodate it or your retry must survive it. Ideally both.

The fix is not just "raise the timeout to 120 seconds", although you will probably do that too. The fix is to make every retry carry the identity of the logical operation, not the identity of the attempt.

``` php
import hashlib, json

def operation_key(payload: dict) -> str:
    # Same logical operation -> same key, across every retry.
    canonical = json.dumps(payload, sort_keys=True, separators=(",", ":"))
    return hashlib.sha256(canonical.encode()).hexdigest()

# before (broken): a fresh key per attempt
for attempt in range(3):
    post(payload, headers={"Idempotency-Key": str(uuid4())})

# after (safe): one key per logical operation
key = operation_key(payload)
for attempt in range(3):
    post(payload, headers={"Idempotency-Key": key})
```

Then decide what each failure class means, and write it down as a table:

| Failure observed | Safe reaction | Unsafe reaction |
|---|---|---|
| Timeout / ack lost | Retry with the same operation key, backoff + jitter | Retry with a fresh key |
| 429 rate limited | Backoff, honor `Retry-After`
|
Immediate retry |
| 5xx server error | Retry with backoff, open a circuit breaker after N | Retry forever |
| 4xx validation error | Dead-letter the message, do not retry | Retry; it will fail identically |

Before you point the real queue at the new endpoint, run both endpoints side by side and compare outcomes.

A canary is only a canary if it has a gate. The gate here is the invariant: after cutover, a retried request must not create a second side effect. If the shadow run shows duplicates, you have not fixed the retry policy; you have only moved it to a cheaper server.

This workflow assumes a batch workload that tolerates a wide tail, and it assumes you control the consumer code. If you are building a chat experience where a human is waiting, a free endpoint's tail will read as "the product is broken", no matter how good the median is. If your side effects are emails, invoices, or writes, and you cannot attach an idempotency key, then no retry policy will save you. And if you need a contractual SLA, a free tier is not a contract; it is an experiment with an excellent price.

MonkeyCode's free model access and free server option are a reasonable place to run this experiment, because the cost of being wrong is zero, which is exactly what an experiment should cost. Treat them as capacity you measure, not capacity you assume, and the free tier stops being a gamble.

So here is the question to answer before you cut over: which event order breaks your invariant? The request that times out, completes server-side, and is retried with a fresh key — does your system reject, replay, or compensate? If you do not know, run the probe, and do not ship until the answer is boring.