{"slug": "five-fail-closed-gates-one-script-rehearse-your-ai-feature-on-a-free-tier-first", "title": "Five Fail-Closed Gates, One Script: Rehearse Your AI Feature on a Free Tier First", "summary": "A developer published a dependency-free Python script, gates.py, that runs five fail-closed checks on a small AI feature's model output — schema validation, output size, latency, token budget, and rollback kill-switch presence — and exits non-zero on any failure. The harness ships with deliberately broken fixtures (empty, garbage, truncated JSON, oversized output) so developers can confirm the gates actually fail before wiring them to a live provider. The author notes the approach runs in about two seconds and can be rehearsed on a free tier, with the caveat that call_live() must still be wired to a real provider.", "body_md": "My readiness checklist for small AI features is a Markdown file. It has nine boxes and zero exit codes. Nobody has opened it since August.\n\nThat is the whole problem. A checklist you read is a wish. A checklist you run is a gate.\n\nSo I rewrote mine as one script. It runs in about two seconds. It exits non-zero when anything is wrong. And it runs first against a deliberately broken fake model, not against a real one.\n\nHere is the harness, the evidence it writes, and the limits of rehearsing on a free tier.\n\nMost teams test the model. Few test the guard around the model.\n\nAsk yourself: if the model returned truncated JSON right now, what happens? Does your service crash, show a blank card, or quietly bill you twice? If you cannot answer in one sentence, you have a doc, not a gate.\n\nA gate needs three parts. A deterministic check. A written evidence record. A fail-closed action that runs without a human.\n\nThis is the step everyone skips. You cannot trust a green run until you have seen a red one.\n\nSo the harness ships with fixtures that are already broken. Empty string. Garbage prose. Truncated JSON. Oversized output.\n\nRun the suite against those fixtures first. If anything passes, your gate is decoration. Fix the gate before you touch a real provider.\n\nThat single habit has saved me more deploy nerves than any prompt tweak.\n\nPure standard library. No dependencies. Drop it in `tools/gates.py`.\n\n``` bash\n#!/usr/bin/env python3\n\"\"\"gates.py - five fail-closed gates for a small AI feature.\"\"\"\nimport argparse\nimport json\nimport os\nimport time\nfrom dataclasses import dataclass, asdict, field\n\nMAX_OUT_CHARS = 4000\nMAX_LATENCY_MS = 4000\nREQUIRED_KEYS = {\"summary\", \"confidence\"}\n\nFIXTURES = {\n    \"empty\": \"\",\n    \"garbage\": \"lorem ipsum dolor sit amet \" * 40,\n    \"truncated\": '{\"summary\": \"refunds take five days\", \"confidence\": 0.',\n    \"oversize\": json.dumps({\"summary\": \"x\" * 9000, \"confidence\": 0.9}),\n    \"good\": json.dumps({\"summary\": \"refunds take five days\", \"confidence\": 0.9}),\n}\n\n@dataclass\nclass Result:\n    gate: str\n    ok: bool\n    detail: str\n    evidence: dict = field(default_factory=dict)\n\ndef call_fake(fixture: str) -> str:\n    return FIXTURES[fixture]\n\ndef call_live(prompt: str) -> str:\n    # Wire this to your provider. Read the key from os.environ only.\n    raise SystemExit(\"call_live() is not wired yet\")\n\ndef gate_schema(text: str) -> Result:\n    try:\n        obj = json.loads(text)\n    except json.JSONDecodeError as e:\n        return Result(\"schema\", False, f\"unparseable: {e}\", {\"chars\": len(text)})\n    missing = REQUIRED_KEYS - obj.keys()\n    return Result(\"schema\", not missing, f\"missing: {sorted(missing)}\", {\"keys\": sorted(obj)})\n\ndef gate_size(text: str) -> Result:\n    ok = 0 < len(text) <= MAX_OUT_CHARS\n    return Result(\"size\", ok, f\"{len(text)} chars, cap {MAX_OUT_CHARS}\")\n\ndef gate_latency(ms: int) -> Result:\n    return Result(\"latency\", ms <= MAX_LATENCY_MS, f\"{ms} ms, ceiling {MAX_LATENCY_MS} ms\")\n\ndef gate_budget(used: int, allowance: int) -> Result:\n    ok = used < allowance * 0.2\n    return Result(\"budget\", ok, f\"{used} of {allowance} tokens, 20% cap\")\n\ndef gate_rollback(flag: str) -> Result:\n    ok = os.path.exists(flag)\n    return Result(\"rollback\", ok, f\"kill switch at {flag}\")\n\ndef main() -> None:\n    ap = argparse.ArgumentParser()\n    ap.add_argument(\"--model\", choices=[\"fake\", \"live\"], default=\"fake\")\n    ap.add_argument(\"--fixture\", default=\"garbage\", choices=sorted(FIXTURES))\n    ap.add_argument(\"--out\", default=\"evidence/gates.json\")\n    args = ap.parse_args()\n\n    allowance = int(os.environ.get(\"TOKEN_ALLOWANCE\", \"0\"))\n    if allowance <= 0:\n        raise SystemExit(\"set TOKEN_ALLOWANCE to your own plan's number\")\n\n    start = time.monotonic()\n    if args.model == \"fake\":\n        text = call_fake(args.fixture)\n    else:\n        text = call_live(\"summarize the support fixture\")\n    ms = int((time.monotonic() - start) * 1000)\n\n    results = [\n        gate_schema(text),\n        gate_size(text),\n        gate_latency(ms),\n        gate_budget(len(text.split()), allowance),\n        gate_rollback(os.environ.get(\"KILL_SWITCH\", \".rollback/feature.flag\")),\n    ]\n\n    failed = [r.gate for r in results if not r.ok]\n    payload = {\n        \"when\": time.strftime(\"%Y-%m-%dT%H:%M:%SZ\", time.gmtime()),\n        \"model\": args.model,\n        \"fixture\": args.fixture,\n        \"passed\": len(results) - len(failed),\n        \"failed\": failed,\n        \"results\": [asdict(r) for r in results],\n    }\n    os.makedirs(os.path.dirname(args.out), exist_ok=True)\n    with open(args.out, \"w\") as f:\n        json.dump(payload, f, indent=2)\n    print(\"failed: \" + \", \".join(failed) if failed else \"all gates passed\")\n    raise SystemExit(1 if failed else 0)\n\nif __name__ == \"__main__\":\n    main()\nmkdir -p .rollback && touch .rollback/feature.flag\nexport TOKEN_ALLOWANCE=100000\npython3 tools/gates.py --model fake --fixture garbage; echo \"exit=$?\"\npython3 tools/gates.py --model fake --fixture good; echo \"exit=$?\"\n```\n\nThe first run must print `failed: schema` and `exit=1`. The second must print `all gates passed` and `exit=0`.\n\nIf both runs look identical, stop. You have a logging script, not a gate.\n\nThen open `evidence/gates.json`. That file is the thing you attach to a release, not a screenshot of a green checkbox.\n\n| Gate | Evidence written | Fail-closed action | \n|---|---|---|\n| schema | raw output plus parse error | serve cached result or a plain error card | \n| size | character count vs cap | re-ask once, then error | \n| latency | measured milliseconds | fall back to the non-AI path | \n| budget | tokens used vs allowance | stop calls for the day, no silent retry | \n| rollback | kill-switch path and mtime | refuse to deploy if the flag is missing | \n\nFive rows. Five verdicts. No adjectives.\n\nHere is where a free option earns its place. Failing paths cost calls. Empty output, truncation, retries, oversized responses — those are the expensive ones to discover in production.\n\nI rehearse the whole red path against MonkeyCode's free model access and free server option. Disclosure: This article was prepared as part of MonkeyCode's product outreach. The free allowance lets me burn calls on intentionally broken fixtures without a card on file.\n\nTwo rules keep this honest.\n\n`TOKEN_ALLOWANCE`. Never hard-code a figure you read in a blog post, including this one.\nA free server is also a fine place to host the kill switch check. If `curl` cannot reach your flag, your rollback is a rumour.\n\nBe blunt with yourself here.\n\nSkip this if your feature already moves money. Skip it if you have a pager rotation and real SLAs; you need runbooks, not scripts. Skip it if there is nobody who can own a red exit code at 2am.\n\nAlso skip it if you cannot name your kill switch. That is a design gap, and no harness fixes it.\n\nGive it two hours. If the fake run still cannot go red, the harness is wrong. Delete it and start over rather than keeping a fuzzy gate that always passes.\n\nAnd if a gate cannot produce a deterministic verdict, remove it. A checklist with nine soft boxes is how we got here.\n\nOne question for my next iteration: what is the one input your feature cannot survive losing? I want to turn that answer into the next fixture.", "url": "https://wpnews.pro/news/five-fail-closed-gates-one-script-rehearse-your-ai-feature-on-a-free-tier-first", "canonical_source": "https://dev.to/rivera123/five-fail-closed-gates-one-script-rehearse-your-ai-feature-on-a-free-tier-first-4n1j", "published_at": "2026-10-20 09:15:15+00:00", "updated_at": "2026-10-20 09:19:05.335498+00:00", "lang": "en", "topics": ["ai-tools", "developer-tools", "mlops", "ai-agents", "ai-safety"], "entities": ["gates.py"], "also_reported_by": [], "alternates": {"html": "https://wpnews.pro/news/five-fail-closed-gates-one-script-rehearse-your-ai-feature-on-a-free-tier-first", "markdown": "https://wpnews.pro/news/five-fail-closed-gates-one-script-rehearse-your-ai-feature-on-a-free-tier-first.md", "text": "https://wpnews.pro/news/five-fail-closed-gates-one-script-rehearse-your-ai-feature-on-a-free-tier-first.txt", "jsonld": "https://wpnews.pro/news/five-fail-closed-gates-one-script-rehearse-your-ai-feature-on-a-free-tier-first.jsonld"}}