Your retry logic has never seen a real 429. Here is how to test it with a local fault-injecting mock

Most LLM client code has a retry branch that has never actually run. The provider returns 200 in every test, so the 429, 500 and timeout paths sit untested until production hits them at 3am.

The fix is cheap: point your client at a local server that fails on a schedule you control.

Run a tiny HTTP server on localhost that mimics a chat-completions endpoint. Give it a failure schedule, for example:

[
  {"status": 429, "retry_after": 2},
  {"status": 500},
  {"status": "timeout"},
  {"status": 200}
]

Each incoming request consumes the next step. Your client sees a 429 with a Retry-After header, then a 500, then a hang, then a good response.

import json, sys, time
from http.server import BaseHTTPRequestHandler, HTTPServer

SCHEDULE = json.load(open(sys.argv[1]))
step = {"i": 0}

class H(BaseHTTPRequestHandler):
    def do_POST(self):
        s = SCHEDULE[min(step["i"], len(SCHEDULE) - 1)]
        step["i"] += 1
        if s["status"] == "timeout":
            time.sleep(60)
            return
        self.send_response(s["status"])
        if "retry_after" in s:
            self.send_header("Retry-After", str(s["retry_after"]))
        self.end_headers()
        self.wfile.write(b'{"choices":[{"message":{"content":"ok"}}]}')

HTTPServer(("127.0.0.1", 9797), H).serve_forever()

Point your client's base URL at http://127.0.0.1:9797 and run your normal code path.

I bundled a slightly more complete version (schedule file, README, examples) as a pay-what-you-want download, minimum $1: https://renevibe76.gumroad.com/l/sargxm

The snippet above is enough to get started on your own, though. The point is to see your retry code run at least once before production does it for you.