{"slug": "cut-over-the-model-path-or-don-t-ship-a-fail-closed-inference-checklist", "title": "Cut Over the Model Path or Don't Ship: A Fail-Closed Inference Checklist", "summary": "An engineer published a fail-closed inference checklist for teams shipping AI features, arguing that a green deploy does not prove production traffic has been cut over from drafting or lab model endpoints. The checklist requires four artifacts before an AI feature leaves the branch: a pinned production base URL matching a committed allowlist, a pinned model id, CI that greps deployable config for denylisted lab and tunnel hosts, and bounded timeouts and retries that never swap the destination. The author warns that retrying a failed production call against a drafting host turns an outage into a data leak, and recommends shipping with the feature flag off until the receipts exist.", "body_md": "If production can still reach the model endpoint you used while drafting, you did not cut over. A green deploy does not prove that. It only proves the process started.\n\nTreat the inference path as a promotion surface. Pin the base URL, the model identity, the timeout, and the fallback. If any of those still point at a lab or shared drafting host, fail the gate. Do not “try prod first and fall back to the sandbox.” That pattern turns an outage into a data leak.\n\nThis is not a vibe-coding essay. It is a copy-paste checklist plus a small policy checker you can run in CI. Use it when an AI feature is about to leave the branch.\n\nYou added a chat box, a summarizer, or a “explain this diff” button. During development the client pointed at whatever answered quickly. That is normal. It is also how lab hosts become silent production dependencies.\n\nThe bug is not “we used a model.” The bug is an unnamed path. Someone pasted a base URL into a helper. A generated client baked it into a default. A retry wrapper treats HTTP 429 from prod as a reason to call the drafting host. None of that shows up in a unit test that mocks `complete()`.\n\nYou need evidence that prod traffic cannot reach the lab path. You also need evidence that when prod is unhealthy, the feature fails closed instead of wandering.\n\nCut over is not “we created a prod API key.” Cut over means all four of these are true at once:\n\n`.env.local`.` default`, `latest`, or an empty string.\nIf you cannot show those four, you are still in the sandbox. Ship the feature flag off.\n\nEvery gate needs an artifact. A Slack “looks good” is not a receipt.\n\n**Pass:** `INFERENCE_BASE_URL` in the production secret store matches a committed allowlist. The value is HTTPS, has no path wildcards, and is not a personal tunnel.\n\n**Fail closed:** any prod manifest that still contains `localhost`, `127.0.0.1`, `ngrok`, `trycloudflare`, or a hostname tagged `lab`, `dev`, `sandbox`, or `draft`.\n\n**Receipt:** the commit SHA of `inference-allowlist.json` plus the secret-store version id.\n\n**Pass:** production sets `INFERENCE_MODEL_ID` to a pinned id your vendor or self-hosted gateway documents. The same id appears in the runbook.\n\n**Fail closed:** `latest`, `auto`, empty, or a name that only exists on the drafting server.\n\n**Receipt:** a one-line contract in the repo: model id, max tokens, and who owns rotation.\n\n**Pass:** CI greps deployable config (Helm, Terraform, Docker Compose prod overlay, sealed secrets templates) and fails on denylisted hosts.\n\n**Fail closed:** denylist bypassed with “temporary” comments, or the check only scans `src/` and ignores `deploy/`.\n\n**Receipt:** the CI job log URL for the merge commit.\n\n**Pass:** production config sets request timeout, connect timeout, max retries, and a per-request token ceiling. Retries do not change the destination.\n\n**Fail closed:** unlimited retries, exponential backoff with no cap, or a retry that swaps `base_url`.\n\n**Receipt:** the config snippet and a test that asserts the client is constructed once with prod settings.\n\n**Pass:** on timeout, 5xx, or quota errors, the feature returns a user-visible failure and an internal metric. No second client.\n\n**Fail closed:** “if prod fails, call the free server so the demo still works.”\n\n**Receipt:** a failing integration test that stubs prod errors and asserts the lab host is never dialed.\n\n**Pass:** production requests send a stable `X-Service` / `User-Agent` and an environment tag `prod`. Logs can answer “which app, which model id, which base URL” without reading source.\n\n**Fail closed:** the drafting client and the prod client are the same binary with the same defaults.\n\n**Receipt:** one redacted log line from a staging call that already uses the prod origin.\n\nUse this in the PR. Check a box only when the artifact exists.\n\n`inference-allowlist.json` lists the single prod origin (or the exact set of regional origins).`inference-denylist.json` lists lab, draft, and tunnel hosts.`INFERENCE_BASE_URL` and `INFERENCE_MODEL_ID`; neither is blank.` base_url` is missing.\nIf a box depends on “we will add it after launch,” the gate failed.\n\nCommit this as `inference-policy.json`. Keep it boring. Boring is reviewable.\n\n```\n{\n  \"allow_base_urls\": [\n    \"https://inference.prod.example.internal\"\n  ],\n  \"deny_host_substrings\": [\n    \"localhost\",\n    \"127.0.0.1\",\n    \"ngrok\",\n    \"trycloudflare\",\n    \"lab.\",\n    \"sandbox.\",\n    \"draft.\",\n    \"dev-inference\"\n  ],\n  \"require_env\": [\n    \"INFERENCE_BASE_URL\",\n    \"INFERENCE_MODEL_ID\",\n    \"INFERENCE_TIMEOUT_MS\",\n    \"INFERENCE_MAX_RETRIES\",\n    \"INFERENCE_MAX_OUTPUT_TOKENS\"\n  ],\n  \"max_retries\": 1,\n  \"forbid_fallback_base_url\": true\n}\n```\n\nReplace the allow URL with yours. Do not add the drafting host “just for staging” in the same file that production loads. Staging gets its own overlay.\n\nThe script below is a proposal you can run locally and in CI. It does not call any model. It only inspects env and text files you pass as deploy roots. Label it unproven against your repo until you execute it once and keep the log.\n\n``` bash\n#!/usr/bin/env python3\n\"\"\"Fail closed if prod config can still reach a lab inference host.\"\"\"\n\nfrom __future__ import annotations\n\nimport json\nimport os\nimport sys\nfrom pathlib import Path\nfrom urllib.parse import urlparse\n\nPOLICY = Path(\"inference-policy.json\")\nSCAN_ROOTS = [Path(\"deploy\"), Path(\"k8s\"), Path(\"infra\"), Path(\".\")]\nSCAN_SUFFIXES = {\".yml\", \".yaml\", \".json\", \".tf\", \".env\", \".toml\"}\n\ndef load_policy() -> dict:\n    if not POLICY.exists():\n        print(\"FAIL: inference-policy.json missing\")\n        sys.exit(2)\n    return json.loads(POLICY.read_text())\n\ndef host_of(url: str) -> str:\n    parsed = urlparse(url if \"://\" in url else f\"https://{url}\")\n    return (parsed.hostname or \"\").lower()\n\ndef denied(host: str, needles: list[str]) -> str | None:\n    for needle in needles:\n        if needle.lower() in host:\n            return needle\n    return None\n\ndef check_env(policy: dict) -> list[str]:\n    errors = []\n    for key in policy[\"require_env\"]:\n        if not os.environ.get(key):\n            errors.append(f\"missing env {key}\")\n    base = os.environ.get(\"INFERENCE_BASE_URL\", \"\")\n    if base:\n        host = host_of(base)\n        hit = denied(host, policy[\"deny_host_substrings\"])\n        if hit:\n            errors.append(f\"INFERENCE_BASE_URL host matches denylist '{hit}': {host}\")\n        allowed_hosts = {host_of(u) for u in policy[\"allow_base_urls\"]}\n        if host not in allowed_hosts:\n            errors.append(f\"INFERENCE_BASE_URL host not allowlisted: {host}\")\n    model = os.environ.get(\"INFERENCE_MODEL_ID\", \"\")\n    if model.lower() in {\"\", \"latest\", \"auto\", \"default\"}:\n        errors.append(f\"INFERENCE_MODEL_ID is not pinned: {model!r}\")\n    try:\n        retries = int(os.environ.get(\"INFERENCE_MAX_RETRIES\", \"99\"))\n    except ValueError:\n        retries = 99\n        errors.append(\"INFERENCE_MAX_RETRIES is not an int\")\n    if retries > int(policy[\"max_retries\"]):\n        errors.append(f\"retries {retries} exceed policy max {policy['max_retries']}\")\n    return errors\n\ndef check_files(policy: dict) -> list[str]:\n    errors = []\n    needles = policy[\"deny_host_substrings\"]\n    for root in SCAN_ROOTS:\n        if not root.exists():\n            continue\n        for path in root.rglob(\"*\"):\n            if not path.is_file() or path.suffix.lower() not in SCAN_SUFFIXES:\n                continue\n            if path.name == POLICY.name:\n                continue\n            # Skip local-only samples; prod overlays must still pass.\n            if \".local.\" in path.name or path.name.endswith(\".example\"):\n                continue\n            text = path.read_text(errors=\"ignore\")\n            lower = text.lower()\n            for needle in needles:\n                if needle.lower() in lower and \"prod\" in path.parts:\n                    errors.append(f\"{path}: denylist hit {needle!r}\")\n    return errors\n\ndef main() -> int:\n    policy = load_policy()\n    errors = check_env(policy) + check_files(policy)\n    if errors:\n        print(\"FAIL-CLOSED: inference cutover incomplete\")\n        for item in errors:\n            print(f\" - {item}\")\n        return 1\n    print(\"PASS: inference path looks cut over (config scan only)\")\n    return 0\n\nif __name__ == \"__main__\":\n    raise SystemExit(main())\n```\n\nWire it so merge is impossible when it exits 1:\n\n```\n# .github/workflows/inference-cutover.yml\nname: inference-cutover\non:\n  pull_request:\n    paths:\n      - \"deploy/**\"\n      - \"k8s/**\"\n      - \"infra/**\"\n      - \"inference-policy.json\"\n      - \".github/workflows/inference-cutover.yml\"\njobs:\n  fail-closed:\n    runs-on: ubuntu-latest\n    steps:\n      - uses: actions/checkout@v4\n      - name: Require prod env in the CI job (from secrets, not the PR)\n        env:\n          INFERENCE_BASE_URL: ${{ secrets.INFERENCE_BASE_URL }}\n          INFERENCE_MODEL_ID: ${{ secrets.INFERENCE_MODEL_ID }}\n          INFERENCE_TIMEOUT_MS: ${{ secrets.INFERENCE_TIMEOUT_MS }}\n          INFERENCE_MAX_RETRIES: ${{ secrets.INFERENCE_MAX_RETRIES }}\n          INFERENCE_MAX_OUTPUT_TOKENS: ${{ secrets.INFERENCE_MAX_OUTPUT_TOKENS }}\n        run: python3 scripts/check_inference_cutover.py\n```\n\nSecrets belong in the store, not in the PR description. If CI has no prod URL, the job must fail. A skipped check is an open gate.\n\n| Symptom | What you might tell yourself | Fail-closed action | \n|---|---|---|\n| Drafting host still in prod overlay | “Staging needs it” | Split overlays. Prod overlay cannot parse the lab hostname. | \n| Model id is `latest` | “We want improvements automatically” | Pin an id. Rotate with a ticket and a replay test. | \n| Prod 429, client calls lab | “Users should still get an answer” | Return an error. Page the owner. Never change `base_url` in retry. | \n| Unit tests mock the SDK | “Coverage is high” | Add one test that inspects the constructed URL. | \n| Tunnel URL in a hotfix | “Only for this incident” | Incident flag off. Do not hot-patch origin. | \n| Free drafting server is fast today | “We can launch on it” | Launch is a contract plus capacity you control. Speed is not a contract. | \n\nDrafting against a shared or free inference endpoint is fine. Shipping that endpoint is not.\n\nDisclosure: This article was prepared as part of MonkeyCode's product outreach.\n\nIf you need a place to sketch clients, prompts, and the checker itself, MonkeyCode’s free model access and free server option can stay on the drafting side of the line. Keep that origin on the denylist for production overlays. Generate the policy file there if you want. Then run the checker against the repo you actually deploy. The product mention ends here because the gate does not care which editor you used. It cares whether prod can still dial the lab.\n\nDo not use this checklist as a substitute for load tests, eval suites, or a real vendor contract. It will not tell you the model is accurate. It will not size GPUs. It will not prove GDPR. It only proves you did not leave a drafting path wired into the live service.\n\nSkip it if you have no production inference origin yet. In that case the honest gate is “do not ship the feature,” not “scan a policy file that allowlists a wish.”\n\nAlso skip it if your app does not call a model at runtime. Generated code that never performs inference is a different review: dependency pins, tests, and side effects. This article is only for the path that answers `complete()` after deploy.\n\nThe scanner is string-based. Obfuscated URLs, runtime service discovery, and hosts injected by a sidecar can evade it. Pair it with egress policy in the mesh or network layer. If the cluster can resolve the lab hostname, add a network deny.\n\nThe allowlist does not prove capacity. A pinned origin can still 429. Fail closed on that too: user-visible error, metric, page. Do not reopen the sandbox to absorb overflow.\n\nAliases will drift. Re-run the job when you rotate model ids. If nobody owns rotation, you do not have a contract. You have a default.\n\nName the production origin. Pin the model id. Deny the lab host in CI. Cap retries without changing destination. Link the job log on the PR. If you cannot, leave the flag off.\n\nThat is the whole method. The drafting environment can be free, shared, or noisy. Production cannot inherit it.", "url": "https://wpnews.pro/news/cut-over-the-model-path-or-don-t-ship-a-fail-closed-inference-checklist", "canonical_source": "https://dev.to/codecpp_5026/cut-over-the-model-path-or-dont-ship-a-fail-closed-inference-checklist-39l9", "published_at": "2026-09-19 09:39:15+00:00", "updated_at": "2026-09-19 09:54:18.141855+00:00", "lang": "en", "topics": ["ai-infrastructure", "mlops", "ai-safety", "developer-tools"], "entities": ["INFERENCE_BASE_URL", "INFERENCE_MODEL_ID", "ngrok", "trycloudflare", "Helm", "Terraform"], "alternates": {"html": "https://wpnews.pro/news/cut-over-the-model-path-or-don-t-ship-a-fail-closed-inference-checklist", "markdown": "https://wpnews.pro/news/cut-over-the-model-path-or-don-t-ship-a-fail-closed-inference-checklist.md", "text": "https://wpnews.pro/news/cut-over-the-model-path-or-don-t-ship-a-fail-closed-inference-checklist.txt", "jsonld": "https://wpnews.pro/news/cut-over-the-model-path-or-don-t-ship-a-fail-closed-inference-checklist.jsonld"}}