Gate Agent Evals by Severity, Not a Flat Pass-Rate A developer built severity_gate.py, an offline tool that gates agent deployments based on failure severity rather than a flat pass-rate. The tool returns SHIP, REVIEW, or BLOCK by reading per-severity distributions, and flipping one severity_class to critical can change the verdict from SHIP to BLOCK while the pass-rate remains unchanged at 92.5%. Gating an agent's eval run by severity means the deploy decision reads the per-severity distribution of the failures, not one flat pass-rate. severity gate.py is an offline, stdlib-only tool that returns SHIP, REVIEW, or BLOCK. In this post's fixtures, flipping one severity class field to critical turns SHIP into BLOCK while the pass-rate holds at 92.5%. AI disclosure:I wrote severity gate.py with an AI assistant and ran it myself, offline, before publishing. Every output block below is pasted from a real local run on Python 3.13.5, standard library only, no network. I checked the exit codes 0 / 1 / 2 , hashed the STDOUT of each scenario twice to confirm it is byte-for-byte deterministic, and edited every line. The 512 / 31 / 6 figures and the "rounding error" line belong to Ethan Walker's Dev.to post, not to me; I link the primary sources and keep their numbers in their own paragraphs, away from my fixture counts 40 / 3 / 2, 92.5% . In short: severity class from low to critical on two of them, and the verdict flips SHIP to BLOCK. The pass-rate does not move. The diff is two lines. json , sys . Offline, keyless, read-only, zero network, deterministic STDOUT. Exit 0 / 1 / 2 for a CI gate. The tool and every fixture are in this post.Here is the ritual I keep seeing at the end of an agent's CI pipeline. The eval suite runs, a number comes out, someone reads it, and if it clears the bar the change ships. The number is a mean: passes over total. It is a fine thing to track . It tells you, roughly, whether the run got better or worse than last week. It is a bad thing to gate on, and the reason is simple arithmetic. A mean gives every failure the same weight. So a run where forty tests failed on trailing whitespace and a run where one test leaked a customer's email into a log both cost the same amount of pass-rate. If your suite is big enough, the second run rounds off to nothing. The one failure that should stop the deploy is invisible at the resolution of the average. That is the whole gap this tool sits in. The pass-rate is tracking : a post-hoc measurement of what happened. The decision about what ships is control , and control has to see the distribution the mean flattens. severity gate.py reads the finished eval results and makes the ship decision on the failures' severity, not their count. The tool keeps the pass-rate on screen, for reference, and then ignores it for the parts of the decision the mean can't reach. Three verdicts, one policy: critical . This fires no matter what the aggregate is. One critical fail out of ten thousand green cases is still a BLOCK. high , or the flat pass-rate dropped below ship threshold . A human looks. It is not an auto-ship and it is not a hard stop.The severity labels are yours. The tool does not invent them and does not guess them; it reads the severity class field you put on each case. That is the load-bearing limit, and I come back to it at the end. What the tool contributes is that once you have committed to a label, the ship decision follows it deterministically instead of dissolving into an average. No keys. No network. No install beyond Python. Save the file, point it at a results JSON, run one command. Here is the whole thing, one file, standard library only: bash /usr/bin/env python3 """ severity gate.py -- an offline deploy gate that reads a FINISHED eval run and decides SHIP / REVIEW / BLOCK from the per-severity distribution of the failures, not from a single flat pass-rate. It reads a JSON file of eval RESULTS it never runs the agent or the eval plus an optional policy file. For every case it reads a verdict pass/fail and the severity class the USER assigned critical / high / medium / low . It reports the flat pass-rate for reference, then makes the ship decision on a rule the mean cannot express: BLOCK -- at least one FAIL whose severity class is in the policy's block on set default: critical . Fires regardless of the aggregate. REVIEW -- no blocking fail, but either a FAIL in review on default: high , or the flat pass-rate is below ship threshold. A human looks; it is not an auto-ship and not a hard block. SHIP -- no blocking fail, no review fail, and pass-rate = ship threshold. The point the tool exists to make: flip one field -- the severity class on a handful of already-failing cases -- and the verdict moves SHIP - BLOCK while the pass-rate does not move at all. The aggregate MEASURES the run; it was never the control over what ships. Offline. Keyless. Read-only. Zero network. Standard library only json, sys . No subprocess, no exec, no eval, no import of the analyzed results, no model, no DB. The fixtures are read with json.load; they are DATA, never executed. Output is byte-for-byte deterministic across runs. It does NOT detect PII, does NOT judge whether a verdict is correct, and does NOT find new failures. It re-weights failures YOU already labeled. Garbage labels in, garbage gate out. It decouples the ship decision from the scalar mean; it does not discover anything. Exit codes usable as a CI gate : 0 SHIP 1 REVIEW or BLOCK the verdict text says which, and why 2 bad input missing / unreadable / unparseable file, missing field, unknown verdict or severity class -- fail-closed Usage: python3 severity gate.py