Five Fail-Closed Gates, One Script: Rehearse Your AI Feature on a Free Tier First

A developer published a dependency-free Python script, gates.py, that runs five fail-closed checks on a small AI feature's model output — schema validation, output size, latency, token budget, and rollback kill-switch presence — and exits non-zero on any failure. The harness ships with deliberately broken fixtures (empty, garbage, truncated JSON, oversized output) so developers can confirm the gates actually fail before wiring them to a live provider. The author notes the approach runs in about two seconds and can be rehearsed on a free tier, with the caveat that call_live() must still be wired to a real provider.

My readiness checklist for small AI features is a Markdown file. It has nine boxes and zero exit codes. Nobody has opened it since August. That is the whole problem. A checklist you read is a wish. A checklist you run is a gate. So I rewrote mine as one script. It runs in about two seconds. It exits non-zero when anything is wrong. And it runs first against a deliberately broken fake model, not against a real one. Here is the harness, the evidence it writes, and the limits of rehearsing on a free tier. Most teams test the model. Few test the guard around the model. Ask yourself: if the model returned truncated JSON right now, what happens? Does your service crash, show a blank card, or quietly bill you twice? If you cannot answer in one sentence, you have a doc, not a gate. A gate needs three parts. A deterministic check. A written evidence record. A fail-closed action that runs without a human. This is the step everyone skips. You cannot trust a green run until you have seen a red one. So the harness ships with fixtures that are already broken. Empty string. Garbage prose. Truncated JSON. Oversized output. Run the suite against those fixtures first. If anything passes, your gate is decoration. Fix the gate before you touch a real provider. That single habit has saved me more deploy nerves than any prompt tweak. Pure standard library. No dependencies. Drop it in tools/gates.py . bash /usr/bin/env python3 """gates.py - five fail-closed gates for a small AI feature.""" import argparse import json import os import time from dataclasses import dataclass, asdict, field MAX OUT CHARS = 4000 MAX LATENCY MS = 4000 REQUIRED KEYS = {"summary", "confidence"} FIXTURES = { "empty": "", "garbage": "lorem ipsum dolor sit amet " 40, "truncated": '{"summary": "refunds take five days", "confidence": 0.', "oversize": json.dumps {"summary": "x" 9000, "confidence": 0.9} , "good": json.dumps {"summary": "refunds take five days", "confidence": 0.9} , } @dataclass class Result: gate: str ok: bool detail: str evidence: dict = field default factory=dict def call fake fixture: str - str: return FIXTURES fixture def call live prompt: str - str: Wire this to your provider. Read the key from os.environ only. raise SystemExit "call live is not wired yet" def gate schema text: str - Result: try: obj = json.loads text except json.JSONDecodeError as e: return Result "schema", False, f"unparseable: {e}", {"chars": len text } missing = REQUIRED KEYS - obj.keys return Result "schema", not missing, f"missing: {sorted missing }", {"keys": sorted obj } def gate size text: str - Result: ok = 0 < len text <= MAX OUT CHARS return Result "size", ok, f"{len text } chars, cap {MAX OUT CHARS}" def gate latency ms: int - Result: return Result "latency", ms <= MAX LATENCY MS, f"{ms} ms, ceiling {MAX LATENCY MS} ms" def gate budget used: int, allowance: int - Result: ok = used < allowance 0.2 return Result "budget", ok, f"{used} of {allowance} tokens, 20% cap" def gate rollback flag: str - Result: ok = os.path.exists flag return Result "rollback", ok, f"kill switch at {flag}" def main - None: ap = argparse.ArgumentParser ap.add argument "--model", choices= "fake", "live" , default="fake" ap.add argument "--fixture", default="garbage", choices=sorted FIXTURES ap.add argument "--out", default="evidence/gates.json" args = ap.parse args allowance = int os.environ.get "TOKEN ALLOWANCE", "0" if allowance <= 0: raise SystemExit "set TOKEN ALLOWANCE to your own plan's number" start = time.monotonic if args.model == "fake": text = call fake args.fixture else: text = call live "summarize the support fixture" ms = int time.monotonic - start 1000 results = gate schema text , gate size text , gate latency ms , gate budget len text.split , allowance , gate rollback os.environ.get "KILL SWITCH", ".rollback/feature.flag" , failed = r.gate for r in results if not r.ok payload = { "when": time.strftime "%Y-%m-%dT%H:%M:%SZ", time.gmtime , "model": args.model, "fixture": args.fixture, "passed": len results - len failed , "failed": failed, "results": asdict r for r in results , } os.makedirs os.path.dirname args.out , exist ok=True with open args.out, "w" as f: json.dump payload, f, indent=2 print "failed: " + ", ".join failed if failed else "all gates passed" raise SystemExit 1 if failed else 0 if name == " main ": main mkdir -p .rollback && touch .rollback/feature.flag export TOKEN ALLOWANCE=100000 python3 tools/gates.py --model fake --fixture garbage; echo "exit=$?" python3 tools/gates.py --model fake --fixture good; echo "exit=$?" The first run must print failed: schema and exit=1 . The second must print all gates passed and exit=0 . If both runs look identical, stop. You have a logging script, not a gate. Then open evidence/gates.json . That file is the thing you attach to a release, not a screenshot of a green checkbox. | Gate | Evidence written | Fail-closed action | |---|---|---| | schema | raw output plus parse error | serve cached result or a plain error card | | size | character count vs cap | re-ask once, then error | | latency | measured milliseconds | fall back to the non-AI path | | budget | tokens used vs allowance | stop calls for the day, no silent retry | | rollback | kill-switch path and mtime | refuse to deploy if the flag is missing | Five rows. Five verdicts. No adjectives. Here is where a free option earns its place. Failing paths cost calls. Empty output, truncation, retries, oversized responses — those are the expensive ones to discover in production. I rehearse the whole red path against MonkeyCode's free model access and free server option. Disclosure: This article was prepared as part of MonkeyCode's product outreach. The free allowance lets me burn calls on intentionally broken fixtures without a card on file. Two rules keep this honest. TOKEN ALLOWANCE . Never hard-code a figure you read in a blog post, including this one. A free server is also a fine place to host the kill switch check. If curl cannot reach your flag, your rollback is a rumour. Be blunt with yourself here. Skip this if your feature already moves money. Skip it if you have a pager rotation and real SLAs; you need runbooks, not scripts. Skip it if there is nobody who can own a red exit code at 2am. Also skip it if you cannot name your kill switch. That is a design gap, and no harness fixes it. Give it two hours. If the fake run still cannot go red, the harness is wrong. Delete it and start over rather than keeping a fuzzy gate that always passes. And if a gate cannot produce a deterministic verdict, remove it. A checklist with nine soft boxes is how we got here. One question for my next iteration: what is the one input your feature cannot survive losing? I want to turn that answer into the next fixture.