Your retry logic has never seen a real 429. Here is how to test it with a local fault-injecting mock

A developer published a small local HTTP server that mimics an LLM chat-completions endpoint and returns a configurable failure schedule, letting engineers exercise retry branches for 429s, 500s and timeouts that normally go untested until production. The roughly 25-line Python script reads a JSON schedule, consumes one step per request, emits a Retry-After header on 429s and sleeps to simulate a hang, and is pointed at by changing the client's base URL to localhost. A more complete version with schedule files, a README and examples is offered as a pay-what-you-want download starting at $1.

Most LLM client code has a retry branch that has never actually run. The provider returns 200 in every test, so the 429 , 500 and timeout paths sit untested until production hits them at 3am. The fix is cheap: point your client at a local server that fails on a schedule you control. Run a tiny HTTP server on localhost that mimics a chat-completions endpoint. Give it a failure schedule, for example: {"status": 429, "retry after": 2}, {"status": 500}, {"status": "timeout"}, {"status": 200} Each incoming request consumes the next step. Your client sees a 429 with a Retry-After header, then a 500, then a hang, then a good response. python import json, sys, time from http.server import BaseHTTPRequestHandler, HTTPServer SCHEDULE = json.load open sys.argv 1 step = {"i": 0} class H BaseHTTPRequestHandler : def do POST self : s = SCHEDULE min step "i" , len SCHEDULE - 1 step "i" += 1 if s "status" == "timeout": time.sleep 60 return self.send response s "status" if "retry after" in s: self.send header "Retry-After", str s "retry after" self.end headers self.wfile.write b'{"choices": {"message":{"content":"ok"}} }' HTTPServer "127.0.0.1", 9797 , H .serve forever Point your client's base URL at http://127.0.0.1:9797 and run your normal code path. I bundled a slightly more complete version schedule file, README, examples as a pay-what-you-want download, minimum $1: https://renevibe76.gumroad.com/l/sargxm https://renevibe76.gumroad.com/l/sargxm The snippet above is enough to get started on your own, though. The point is to see your retry code run at least once before production does it for you.