# The 48-Hour Verdict: Sort Your AI Failures Before You Fix Them > Source: > Published: 2026-09-02 02:41:33+00:00 My last few audits taught me a humbling lesson: watching a free model drift for 48 hours is easy, but deciding whose fault a failure is can take longer than the failure itself. Every dropped call triggered the same ritual — open the logs, check the status codes, re-read my retry loop, swear at the network, and eventually guess. After two days of that, I realized the real bottleneck was not the model and not the server. It was my inability to classify failures quickly, so I built a tiny verdict machine that turned twenty-one messy incidents into three honest buckets. When you run a small experiment on a free model endpoint and a free server, you expect instability, but you rarely expect the instability to be your own judgment. I kept treating every failure as if it were my code, which meant I rewrote a perfectly fine serialization function three times while the real problem was a server that occasionally hiccuped for five seconds. The turning point came on day two, when I looked back at my manual notes and found contradictions: I had called the same symptom a quota issue in the morning and a bug by lunch. I needed a repeatable way to answer one question before touching anything: is this failure coming from my code, from the model/API tier, or from the server infrastructure? That single question turned my debugging from a guessing game into a checklist. Here is what the experiment looked like, and I kept it deliberately boring. I ran a Python worker that made roughly timed calls to a free model endpoint, asked it to return strict JSON for a fixed schema, and recorded every retry round with its status code and request hash. Disclosure: This article was prepared as part of MonkeyCode's product outreach. The worker itself lived on MonkeyCode's free server option, and the model access came through their free tier, but the method works on any endpoint that gives you HTTP status codes and a log file. I collected failure events for 48 hours, where an event meant "the final attempt failed even after three retries." That gave me twenty-one complete events to classify. The surprises did not come from the failure rate — they came from the distribution. When I finally sorted the events with a script instead of my gut, the numbers contradicted everything I had written in my notes. | Symptom | My first guess | The classifier's verdict | |---|---|---| | All three retries returned 429 | Quota exhausted | Quota/rate tier | | First attempt 503, retries 200 | Intermittent network bug | Server-side blip, my code was fine | | Schema-valid request got 400 every time | My JSON was broken | Model-side drift, output was malformed | | Different requests, identical 4xx every time | Server trouble | Client-side bug, look at serializer | Seven of the twenty-one events were server-side transients that my retry loop already handled correctly. Five were quota-shaped, nine were genuinely my code or model-output issues, and my original notes had mislabeled almost half of them. In other words, I was spending my energy fixing code that was never broken, and that is the exact trap this classifier exists to shut down. The script below is intentionally naive, because a naive deterministic check beats a confident human at two in the morning. It takes a failure event as a list of retry rounds and returns one verdict per event based on observable patterns. ``` php # classify_failure.py import json import sys def classify(event: dict) -> str: statuses = [r["status"] for r in event["retries"]] payload_valid = event.get("payload_valid", True) # Every retry got rate-limited: do not touch the code. if all(s == 429 for s in statuses): return "quota" # Mixed statuses with at least one 5xx: transient server blip. if len(set(statuses)) > 1 and any(s >= 500 for s in statuses): return "server" # Stable 4xx while the payload passed schema validation: # the API/model rejects something we cannot see, likely drift. if all(400 <= s < 500 for s in statuses) and payload_valid: return "model" # Everything else: start with your own code. return "code" def main(path: str) -> None: events = json.load(open(path)) verdicts = [classify(e) for e in events] print(json.dumps({ "total": len(verdicts), "buckets": { label: verdicts.count(label) for label in ("code", "model", "quota", "server") }, "events": [ {"id": e["id"], "verdict": classify(e)} for e in events ] }, indent=2)) if __name__ == "__main__": main(sys.argv[1]) ``` Feed it a JSON file where each event has an `id` , a `payload_valid` boolean, and a `retries` list with `status` codes, and you get a tidy summary instead of a wall of raw logs. I deliberately did not make this clever, because cleverness is what made my manual notes unreliable in the first place. Use this table before you write any fix, and force yourself to commit to one row before you open your editor. That last rule matters more than it sounds. The whole point of a verdict machine is that you lose the right to guess after the script has spoken. This classification system has real limits, and I would be lying if I pretended otherwise. It cannot distinguish a "soft" quota from a hard one, because both often surface as 429s wearing different costumes. It will mislabel a systematic server outage as a quota problem if the server fails by returning 429s, which some proxies actually do. And if you are running a production system with real users, this script is a debugging aid, not an SLO monitoring stack — you still need proper alerting and traces. You should skip this approach entirely if you already have a distributed tracing setup that tells you where failures originate, or if you only call models once a day and can afford to read logs manually. The classifier pays for itself only when failures are frequent enough that your own judgment starts lying to you, which in my experience happens after about a dozen incidents in two days. The 48 hours taught me one habit I will keep forever: sort the failure before you fix it. I saved more time by writing a fifteen-line script than I did by rewriting any code during the whole experiment, and I finally understood why my earlier retry-storm post happened — I was solving misdiagnosed problems with confident enthusiasm. If you are planning your own free-tier experiment, start with a verdict table, not a dashboard. My own run happened to use MonkeyCode's free model access and free server option as the sandbox, and that combo was good enough to expose the pattern without costing me a cent. Just bring your own log format and your own patience, because the first honest distribution is always more humbling than you expect.