438 of 536 quarantined — and not one was a bad verdict

A developer built an AI agent for a humanitarian NGO to screen payments against OFAC sanctions lists, but a bug caused 438 of 536 counterparties to be quarantined due to rate limits being misclassified as model failures. The fix distinguishes transient errors from genuine adjudication failures, ensuring quarantine remains a meaningful signal for human compliance officers.

A 12-person humanitarian NGO carries exactly the same strict-liability sanctions exposure as JPMorgan, and cannot hire anyone to manage it. I built an agent that does — and the first real run against Gemini quarantined 438 of 536 counterparties. Not one of them was a bad verdict. I created this piece for the purposes of entering the All Things Agentic Hackathon. Repo: https://github.com/edycutjong/interdict · 3-minute demo: Interdict re-screens an NGO's whole payment book whenever Treasury updates the OFAC sanctions list. A true hit gets held — money stops. A lookalike gets cleared with a written reason. When the model's answer can't be trusted, the counterparty goes to quarantine , which is a terminal state: a human compliance officer is told the system could not safely decide, and the money stays frozen until they rule. That's expensive by design. Quarantine is supposed to be rare and it's supposed to mean something. So when 438 of 536 landed there, my first assumption was that the adjudicator had gone haywire. It hadn't. Free-tier Gemini allows five requests a minute. Every call after the first twenty-one came back 429 RESOURCE EXHAUSTED , and my code did this: try: verdict = adjudicator.adjudicate context except Exception as exc: quarantine match id, "PARSE ERROR", {"error": str exc } A rate limit is not a parse error. But except Exception doesn't know that, so 438 transient network conditions were filed as suspected model-integrity failures . The rate limit costs thirty seconds of waiting. The bug costs the escalation queue. Quarantine only works if a human reads it. Put 438 entries in there that needed nothing but patience, and the one entry that genuinely needs a person — a near-identical name where the model's rationale doesn't hold up — is buried underneath them. The queue stops being a signal and becomes noise, and the operator learns to skim it. That's the actual failure, and it would have survived into production looking like a working system. The insight that fixed it is boring and, I think, general: "The model was wrong" and "the model did not answer" are different failures.One is fixed by a human reading the evidence. The other is fixed by waiting. Conflating them means you cannot triage. So I stopped conflating them. First, the adjudicator owns its own backoff, and only retries things that are actually transient: TRANSIENT MARKERS = "RESOURCE EXHAUSTED", "429", "503", "UNAVAILABLE", "DEADLINE EXCEEDED" def is transient exc: Exception - bool: return any m in str exc for m in TRANSIENT MARKERS def retry delay exc: Exception, attempt: int - float: """Seconds to wait. Prefers the server's own hint over our guess.""" m = re.search r"retry in \d+ ?:\.\d+ ? s", str exc if m: return min float m.group 1 + 1.0, 120.0 return min DEFAULT BACKOFF S 2 attempt - 1 , 120.0 That retry in Ns hint matters more than the exponential fallback. The server knows when it will serve you again; guessing is strictly worse than reading. Five attempts, honouring the hint, and only then does it give up — as a distinct exception type : except Exception as exc: if not is transient exc : raise a bad answer is not a slow answer if attempt == MAX TRANSIENT RETRIES: raise TransientAdjudicationError f"model unreachable after {attempt} attempts: {exc}" from exc time.sleep retry delay exc, attempt Then the orchestrator — the only component allowed to write a decision — routes on that type: try: verdict = adjudicator.adjudicate context, feedback=feedback except TransientAdjudicationError as exc: Still quarantine -- money must never move on a decision that was never made -- but say so accurately. quarantine conn, match id, "ADJUDICATOR UNAVAILABLE", { "counterparty id": counterparty id, "error": str exc :500 , "attempt": attempt, "retryable": True, } except Exception as exc: A model failure must never become a silent CLEAR. quarantine conn, match id, "PARSE ERROR", { "counterparty id": counterparty id, "error": str exc :500 , "attempt": attempt, "retryable": False, } Note what did not change: both paths still quarantine, and money still stops in both. The safety property is identical. What changed is that the row now carries retryable: true or retryable: false , so an operator can tell at a glance which pile is which — and the retryable pile drains itself on the next pass without anyone touching it. The distinction is worth more than the retry. If I'd only added backoff, the 438 would have shrunk but the category error would still be there, waiting for the next outage. Three things, in order of how much they cost me: A bare except around a model call is a category error, not a style problem. Model calls fail in at least two ways that demand opposite responses. Any handler that can't distinguish them will eventually make the wrong one, and it will do so quietly. Let the failure type carry the triage. retryable: true|false in the payload is what makes the queue readable. The alternative is an operator reading 438 stack traces to work out which ones matter. Escalation is a budget. Every entry you send to a human spends attention you'll need later. I now treat "should this really escalate?" as a design question with a cost attached, the same way I'd treat a database write. It's a hackathon build, and it's specific about what it isn't: The screening numbers, for what they're worth: top-1 0.995 against an independent oracle's 0.840 , measured on 400 names deliberately perturbed so none appear on the list verbatim. Screening the seeded book verbatim scores 1.000, which is a string-equality test wearing a costume, so I don't report it. Everything above reproduces with make reproduce . If the escalation-budget idea is useful to you, that's the part I'd steal. I created this piece of content for the purposes of entering the All Things Agentic Hackathon. Repo: https://github.com/edycutjong/interdict https://github.com/edycutjong/interdict · Demo: https://youtu.be/C1VFGSwS7w4 https://youtu.be/C1VFGSwS7w4