Generated code looks cheap until the assumptions land. You own every invented config, schema, and side effect. I now treat silent model assumptions as merge blockers.
Did the model ask about your auth scheme? Did it confirm the table names you actually have? If not, you did not get a patch. You got a guess.
This is a catalog, not a pep talk. Each anti-pattern has a symptom, a root cause, and a replacement. Then I give you a constraint file and a gate script you can run on a staged diff.
Agents fill gaps because that is their job. Gap filling helps during a short spike. It becomes poison inside a brownfield service.
Cheap tokens make the guesses faster. They do not make the guesses true. Technical debt arrives as confident Python.
I keep a constraints file next to the repo. The model must read it before it writes. A small gate fails the patch when it invents facts.
Symptom
The patch reads PAYMENT_WEBHOOK_SECRET
on boot. Your secrets manager has never heard of it. Staging dies inside os.environ[...]
.
Root cause
The model completed a typical tutorial shape. Tutorials always hide one missing env var. Nobody listed the real allowlist.
Replacement
Publish an env allowlist in the repo. Reject new keys unless a human adds them.
env_allowlist:
- DATABASE_URL
- REDIS_URL
- APP_ENV
forbidden_env_prefixes:
- AWS_
- STRIPE_
- OPENAI_
Ask yourself: who named that variable? If the answer is "the model," delete it.
Symptom
requirements.txt
gains httpx
or orjson
. Sometimes the name does not even exist. CI installs it, or CI cannot install it.
Root cause
The model optimized for a blog-post stack. It did not read your lockfile. Convenience beat your supply-chain rules.
Replacement
Diff imports against the lockfile. Unknown import roots fail the gate. Humans add libraries on purpose.
ALLOWED_IMPORT_ROOTS = {"flask", "sqlalchemy", "redis", "pydantic"}
Would you merge a mystery wheel from the internet? Then do not merge a mystery import either.
Symptom
The new endpoint has no auth decorator. Or it checks a header the gateway never sets. Or it trusts user_id
from the JSON body.
Root cause
Demos skip auth to keep the snippet small. The model learned those demos. Your threat model never traveled in the prompt.
Replacement
State the auth contract in constraints. Every HTTP handler must match one pattern. No pattern, no merge.
http:
must_use_decorator: "require_session"
forbid_body_fields_as_identity:
- user_id
- account_id
- is_admin
Can an anonymous caller hit this route? If you cannot answer, the patch is incomplete.
Symptom
The query selects users.uuid
. Your table has users.id
. Or the patch adds metadata
JSON nobody migrated.
Root cause
Language models remember popular schemas. They do not remember yours. A plausible column is still a lie.
Replacement
Check identifiers against a schema dump. I keep schema/tables.txt
generated from migrations. Unknown columns fail the same way unknown env fails.
users.id
users.email
users.created_at
orders.id
orders.user_id
orders.total_cents
Did you run the migration, or did the model imagine it? Imagination is not a migration.
Symptom
The helper writes /tmp/cache.json
. It shells out to curl
. It logs access tokens at INFO.
Root cause
The model "finished" the function. Finishing is not the same as isolating. Side effects feel like completeness.
Replacement
Ban whole families of calls in generated diffs. Allow them only in named modules. Keep the blast radius tiny.
side_effects:
forbid_substrings:
- "subprocess."
- "os.system("
- "pathlib.Path('/tmp"
- "open('/tmp"
forbid_log_names:
- password
- token
- authorization
If a spike needs /tmp
, put it in scratch/
. Do not let it ride into app/
.
Here is a compact patch I would reject on sight. It looks helpful. It is five anti-patterns in one function.
+ @app.route("/refunds", methods=["POST"])
+ def refunds():
+ key = os.environ["STRIPE_KEY"]
+ import requests
+ user_id = request.json["user_id"]
+ row = db.execute("SELECT uuid FROM users WHERE id=%s", user_id)
+ open("/tmp/refunds.log", "a").write(str(request.json))
+ return {"ok": True, "user_id": user_id}
What did the model invent? A secret name. A new HTTP client. Identity from the body. A column you do not have. A world-readable temp log.
The gate below should print failures, not a green check. If it passes this diff, your allowlists are too wide.
Here is a small checker you can copy. It is a heuristic, not a full program analysis. Label it as a merge gate, not a proof.
Save constraints.yml
at the repo root. Save this script as tools/assumption_gate.py
. Feed it a unified diff from the model.
#!/usr/bin/env python3
"""Fail a generated diff that invents facts.
This is a proposed gate. Run it on your own diffs.
It does not execute the patch. It only scans added text.
"""
from __future__ import annotations
import argparse
import re
import sys
from pathlib import Path
import yaml
ENV_RE = re.compile(r"os\.environ(?:\[|\.get\()['\"]([A-Z0-9_]+)")
IMPORT_RE = re.compile(r"^(?:from|import)\s+([a-zA-Z0-9_\.]+)", re.M)
IDENT_RE = re.compile(r"\b([a-z_][a-z0-9_]*)\.([a-z_][a-z0-9_]*)\b")
ROUTE_RE = re.compile(r"@app\.route|@router\.")
def load_constraints(path: Path) -> dict:
data = yaml.safe_load(path.read_text())
if not isinstance(data, dict):
raise ValueError("constraints.yml must be a mapping")
return data
def added_lines(diff_text: str) -> str:
lines = []
for line in diff_text.splitlines():
if line.startswith("+") and not line.startswith("+++"):
lines.append(line[1:])
return "\n".join(lines)
def main() -> int:
parser = argparse.ArgumentParser()
parser.add_argument("--diff", required=True)
parser.add_argument("--constraints", default="constraints.yml")
parser.add_argument("--schema", default="schema/tables.txt")
args = parser.parse_args()
constraints = load_constraints(Path(args.constraints))
diff = Path(args.diff).read_text(encoding="utf-8")
added = added_lines(diff)
failures: list[str] = []
allow_env = set(constraints.get("env_allowlist", []))
for key in ENV_RE.findall(added):
if key not in allow_env:
failures.append(f"phantom env: {key}")
allowed_imports = set(constraints.get("allowed_import_roots", []))
for raw in IMPORT_RE.findall(added):
root = raw.split(".")[0]
if allowed_imports and root not in allowed_imports:
failures.append(f"unapproved import: {root}")
schema_path = Path(args.schema)
if schema_path.exists():
allowed_cols = {
tuple(line.strip().split(".", 1))
for line in schema_path.read_text().splitlines()
if "." in line
}
known_tables = {table for table, _ in allowed_cols}
for table, col in IDENT_RE.findall(added):
if table in known_tables and (table, col) not in allowed_cols:
failures.append(f"invented column: {table}.{col}")
for blob in constraints.get("side_effects", {}).get("forbid_substrings", []):
if blob in added:
failures.append(f"side effect: {blob!r}")
decorator = constraints.get("http", {}).get("must_use_decorator")
if decorator and ROUTE_RE.search(added) and decorator not in added:
failures.append("route without auth decorator")
identity_fields = constraints.get("http", {}).get(
"forbid_body_fields_as_identity", []
)
for field in identity_fields:
if re.search(rf"json\[['"]{field}['"]\]", added):
failures.append(f"body used as identity: {field}")
if not failures:
print("assumption gate: pass")
return 0
print("assumption gate: fail")
for item in failures:
print(f" - {item}")
return 1
if __name__ == "__main__":
sys.exit(main())
Run it like this:
git diff --staged > /tmp/staged.diff
python tools/assumption_gate.py --diff /tmp/staged.diff
No staged diff? Pipe the model output through diff -u /dev/null
. The gate still sees every added line. That is enough to catch the five patterns above.
Expected output on the refunds example:
assumption gate: fail
- phantom env: STRIPE_KEY
- unapproved import: requests
- invented column: users.uuid
- side effect: "open('/tmp"
- route without auth decorator
- body used as identity: user_id
If that list is empty, the gate is not wired. Fix the constraints before you blame the model.
| If the diff... | Treat it as | Human action |
|---|---|---|
| Adds an env key | Phantom config | Add to allowlist or delete |
| Adds an import root | Unapproved dependency | Lockfile first, then code |
| Adds a route, no decorator | Happy-path auth | Wrap or reject |
Uses unknown table.col |
||
| Invented schema | Dump schema, then rewrite | |
Touches /tmp or subprocess |
||
| Invisible side effect | Move to scratch/ or drop |
Print this table in the PR template. Reviewers stop arguing taste. They argue facts.
Do not ask for "a refunds endpoint." Ask for a diff that obeys the file. Keep the prompt boring and strict.
Read constraints.yml and schema/tables.txt.
Return a unified diff only.
Do not add env keys outside env_allowlist.
Do not add import roots outside allowed_import_roots.
Do not invent columns.
Every new route must use require_session.
If a fact is missing, ask a question. Do not guess.
Then paste the gate failures back. The second turn should shrink, not sprawl. If it sprawls, the model is still filling gaps. Stop and edit by hand.
I want the model to propose code. I do not want it to propose reality. Those are different jobs.
Disclosure: This article was prepared as part of MonkeyCode's product outreach. I run the generate-then-gate loop on MonkeyCode's free models and free server. The server is a lab for the checker, not a factory for production traffic.
The loop is boring on purpose:
assumption_gate.py
.Free model access makes the retries cheap. The gate makes the retries honest. Without the gate, cheap retries just multiply debt.
This gate reads text. It does not run tests. It will miss a renamed import alias. It will miss SQL built with f-strings.
Object attributes look like columns. user.email
can false-positive when users.email
is real. Keep the schema file on table names, not on instance names. Review those hits instead of auto-fixing them.
It can also nag on legitimate new columns. That is the point of a human allowlist. A noisy fail is better than a silent schema lie.
Do not call this a security audit. Do not skip unit tests because the gate passed. Do not point the lab server at production databases.
Regex will rot as your framework changes. Budget an hour when you upgrade the web layer. Update the decorator name. Update the import roots.
Skip this if you have no lockfile. Skip this if you cannot dump schema. Skip this if the repo is a throwaway spike.
Also skip it for generated front-end CSS churn. The patterns above target service code. A linter war on class names helps nobody.
If you cannot review the allowlists, stop. An outdated allowlist becomes a rubber stamp. Rubber stamps are how assumptions sneak back.
I want candidate diffs. I want them small. I want every new fact to be named.
Ask the model: which constraints did you use? Ask it: which facts did you invent anyway? If it cannot list them, distrust the patch.
Cheap code is a throughput trick. Assumption control is the actual engineering. Keep the catalog next to the gate, not in a wiki.