If production can still reach the model endpoint you used while drafting, you did not cut over. A green deploy does not prove that. It only proves the process started.
Treat the inference path as a promotion surface. Pin the base URL, the model identity, the timeout, and the fallback. If any of those still point at a lab or shared drafting host, fail the gate. Do not “try prod first and fall back to the sandbox.” That pattern turns an outage into a data leak.
This is not a vibe-coding essay. It is a copy-paste checklist plus a small policy checker you can run in CI. Use it when an AI feature is about to leave the branch.
You added a chat box, a summarizer, or a “explain this diff” button. During development the client pointed at whatever answered quickly. That is normal. It is also how lab hosts become silent production dependencies.
The bug is not “we used a model.” The bug is an unnamed path. Someone pasted a base URL into a helper. A generated client baked it into a default. A retry wrapper treats HTTP 429 from prod as a reason to call the drafting host. None of that shows up in a unit test that mocks complete().
You need evidence that prod traffic cannot reach the lab path. You also need evidence that when prod is unhealthy, the feature fails closed instead of wandering.
Cut over is not “we created a prod API key.” Cut over means all four of these are true at once:
.env.local. default, latest, or an empty string.
If you cannot show those four, you are still in the sandbox. Ship the feature flag off.
Every gate needs an artifact. A Slack “looks good” is not a receipt.
Pass: INFERENCE_BASE_URL in the production secret store matches a committed allowlist. The value is HTTPS, has no path wildcards, and is not a personal tunnel.
Fail closed: any prod manifest that still contains localhost, 127.0.0.1, ngrok, trycloudflare, or a hostname tagged lab, dev, sandbox, or draft.
Receipt: the commit SHA of inference-allowlist.json plus the secret-store version id.
Pass: production sets INFERENCE_MODEL_ID to a pinned id your vendor or self-hosted gateway documents. The same id appears in the runbook.
Fail closed: latest, auto, empty, or a name that only exists on the drafting server.
Receipt: a one-line contract in the repo: model id, max tokens, and who owns rotation.
Pass: CI greps deployable config (Helm, Terraform, Docker Compose prod overlay, sealed secrets templates) and fails on denylisted hosts.
Fail closed: denylist bypassed with “temporary” comments, or the check only scans src/ and ignores deploy/.
Receipt: the CI job log URL for the merge commit.
Pass: production config sets request timeout, connect timeout, max retries, and a per-request token ceiling. Retries do not change the destination.
Fail closed: unlimited retries, exponential backoff with no cap, or a retry that swaps base_url.
Receipt: the config snippet and a test that asserts the client is constructed once with prod settings.
Pass: on timeout, 5xx, or quota errors, the feature returns a user-visible failure and an internal metric. No second client.
Fail closed: “if prod fails, call the free server so the demo still works.”
Receipt: a failing integration test that stubs prod errors and asserts the lab host is never dialed.
Pass: production requests send a stable X-Service / User-Agent and an environment tag prod. Logs can answer “which app, which model id, which base URL” without reading source.
Fail closed: the drafting client and the prod client are the same binary with the same defaults.
Receipt: one redacted log line from a staging call that already uses the prod origin.
Use this in the PR. Check a box only when the artifact exists.
inference-allowlist.json lists the single prod origin (or the exact set of regional origins).inference-denylist.json lists lab, draft, and tunnel hosts.INFERENCE_BASE_URL and INFERENCE_MODEL_ID; neither is blank. base_url is missing.
If a box depends on “we will add it after launch,” the gate failed.
Commit this as inference-policy.json. Keep it boring. Boring is reviewable.
{
"allow_base_urls": [
"https://inference.prod.example.internal"
],
"deny_host_substrings": [
"localhost",
"127.0.0.1",
"ngrok",
"trycloudflare",
"lab.",
"sandbox.",
"draft.",
"dev-inference"
],
"require_env": [
"INFERENCE_BASE_URL",
"INFERENCE_MODEL_ID",
"INFERENCE_TIMEOUT_MS",
"INFERENCE_MAX_RETRIES",
"INFERENCE_MAX_OUTPUT_TOKENS"
],
"max_retries": 1,
"forbid_fallback_base_url": true
}
Replace the allow URL with yours. Do not add the drafting host “just for staging” in the same file that production loads. Staging gets its own overlay.
The script below is a proposal you can run locally and in CI. It does not call any model. It only inspects env and text files you pass as deploy roots. Label it unproven against your repo until you execute it once and keep the log.
#!/usr/bin/env python3
"""Fail closed if prod config can still reach a lab inference host."""
from __future__ import annotations
import json
import os
import sys
from pathlib import Path
from urllib.parse import urlparse
POLICY = Path("inference-policy.json")
SCAN_ROOTS = [Path("deploy"), Path("k8s"), Path("infra"), Path(".")]
SCAN_SUFFIXES = {".yml", ".yaml", ".json", ".tf", ".env", ".toml"}
def load_policy() -> dict:
if not POLICY.exists():
print("FAIL: inference-policy.json missing")
sys.exit(2)
return json.loads(POLICY.read_text())
def host_of(url: str) -> str:
parsed = urlparse(url if "://" in url else f"https://{url}")
return (parsed.hostname or "").lower()
def denied(host: str, needles: list[str]) -> str | None:
for needle in needles:
if needle.lower() in host:
return needle
return None
def check_env(policy: dict) -> list[str]:
errors = []
for key in policy["require_env"]:
if not os.environ.get(key):
errors.append(f"missing env {key}")
base = os.environ.get("INFERENCE_BASE_URL", "")
if base:
host = host_of(base)
hit = denied(host, policy["deny_host_substrings"])
if hit:
errors.append(f"INFERENCE_BASE_URL host matches denylist '{hit}': {host}")
allowed_hosts = {host_of(u) for u in policy["allow_base_urls"]}
if host not in allowed_hosts:
errors.append(f"INFERENCE_BASE_URL host not allowlisted: {host}")
model = os.environ.get("INFERENCE_MODEL_ID", "")
if model.lower() in {"", "latest", "auto", "default"}:
errors.append(f"INFERENCE_MODEL_ID is not pinned: {model!r}")
try:
retries = int(os.environ.get("INFERENCE_MAX_RETRIES", "99"))
except ValueError:
retries = 99
errors.append("INFERENCE_MAX_RETRIES is not an int")
if retries > int(policy["max_retries"]):
errors.append(f"retries {retries} exceed policy max {policy['max_retries']}")
return errors
def check_files(policy: dict) -> list[str]:
errors = []
needles = policy["deny_host_substrings"]
for root in SCAN_ROOTS:
if not root.exists():
continue
for path in root.rglob("*"):
if not path.is_file() or path.suffix.lower() not in SCAN_SUFFIXES:
continue
if path.name == POLICY.name:
continue
if ".local." in path.name or path.name.endswith(".example"):
continue
text = path.read_text(errors="ignore")
lower = text.lower()
for needle in needles:
if needle.lower() in lower and "prod" in path.parts:
errors.append(f"{path}: denylist hit {needle!r}")
return errors
def main() -> int:
policy = load_policy()
errors = check_env(policy) + check_files(policy)
if errors:
print("FAIL-CLOSED: inference cutover incomplete")
for item in errors:
print(f" - {item}")
return 1
print("PASS: inference path looks cut over (config scan only)")
return 0
if __name__ == "__main__":
raise SystemExit(main())
Wire it so merge is impossible when it exits 1:
name: inference-cutover
on:
pull_request:
paths:
- "deploy/**"
- "k8s/**"
- "infra/**"
- "inference-policy.json"
- ".github/workflows/inference-cutover.yml"
jobs:
fail-closed:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
- name: Require prod env in the CI job (from secrets, not the PR)
env:
INFERENCE_BASE_URL: ${{ secrets.INFERENCE_BASE_URL }}
INFERENCE_MODEL_ID: ${{ secrets.INFERENCE_MODEL_ID }}
INFERENCE_TIMEOUT_MS: ${{ secrets.INFERENCE_TIMEOUT_MS }}
INFERENCE_MAX_RETRIES: ${{ secrets.INFERENCE_MAX_RETRIES }}
INFERENCE_MAX_OUTPUT_TOKENS: ${{ secrets.INFERENCE_MAX_OUTPUT_TOKENS }}
run: python3 scripts/check_inference_cutover.py
Secrets belong in the store, not in the PR description. If CI has no prod URL, the job must fail. A skipped check is an open gate.
| Symptom | What you might tell yourself | Fail-closed action |
|---|---|---|
| Drafting host still in prod overlay | “Staging needs it” | Split overlays. Prod overlay cannot parse the lab hostname. |
Model id is latest |
“We want improvements automatically” | Pin an id. Rotate with a ticket and a replay test. |
| Prod 429, client calls lab | “Users should still get an answer” | Return an error. Page the owner. Never change base_url in retry. |
| Unit tests mock the SDK | “Coverage is high” | Add one test that inspects the constructed URL. |
| Tunnel URL in a hotfix | “Only for this incident” | Incident flag off. Do not hot-patch origin. |
| Free drafting server is fast today | “We can launch on it” | Launch is a contract plus capacity you control. Speed is not a contract. |
Drafting against a shared or free inference endpoint is fine. Shipping that endpoint is not.
Disclosure: This article was prepared as part of MonkeyCode's product outreach.
If you need a place to sketch clients, prompts, and the checker itself, MonkeyCode’s free model access and free server option can stay on the drafting side of the line. Keep that origin on the denylist for production overlays. Generate the policy file there if you want. Then run the checker against the repo you actually deploy. The product mention ends here because the gate does not care which editor you used. It cares whether prod can still dial the lab.
Do not use this checklist as a substitute for load tests, eval suites, or a real vendor contract. It will not tell you the model is accurate. It will not size GPUs. It will not prove GDPR. It only proves you did not leave a drafting path wired into the live service.
Skip it if you have no production inference origin yet. In that case the honest gate is “do not ship the feature,” not “scan a policy file that allowlists a wish.”
Also skip it if your app does not call a model at runtime. Generated code that never performs inference is a different review: dependency pins, tests, and side effects. This article is only for the path that answers complete() after deploy.
The scanner is string-based. Obfuscated URLs, runtime service discovery, and hosts injected by a sidecar can evade it. Pair it with egress policy in the mesh or network layer. If the cluster can resolve the lab hostname, add a network deny.
The allowlist does not prove capacity. A pinned origin can still 429. Fail closed on that too: user-visible error, metric, page. Do not reopen the sandbox to absorb overflow.
Aliases will drift. Re-run the job when you rotate model ids. If nobody owns rotation, you do not have a contract. You have a default.
Name the production origin. Pin the model id. Deny the lab host in CI. Cap retries without changing destination. Link the job log on the PR. If you cannot, leave the flag off.
That is the whole method. The drafting environment can be free, shared, or noisy. Production cannot inherit it.