Oracle First: Routing AI-Generated Diffs With a Glossary and Four Leaves

An engineer proposes a routing framework for AI-generated code diffs, introducing a glossary and a decision tree to classify patches as keep, quarantine, or rewrite. The approach includes a heuristic classifier script that flags architecture touches, secret hints, and silent expansion, aiming to reduce the cost of reviewing cheaply generated code.

Consider this scene. It is a composite, not a personal war story. An agent ran overnight on a leftover prompt. Morning git status showed fourteen files. Two of them implemented the requested endpoint. The rest were a new logger, a renamed helper, a rewritten Dockerfile, and a README that now contradicted the tests. Generation cost was close to zero. The next four hours were not. That gap is the actual product problem. When a patch is cheap to produce, the expensive work is routing: keep, quarantine, or rewrite. Skip the routing and the cheap code becomes expensive debt with extra files attached. This article is a glossary, a routing tree, and a worked example at each leaf. The artifact is a small classifier plus a quarantine command sequence. Treat the code as a proposal unless you run it on your own repo. Free-model loops optimize for “a diff appeared.” Reviewers optimize for “this diff is safe to merge.” Those are different objective functions. A green unit test on a helper you did not ask for is not evidence that the architecture still holds. Cheap generation also changes failure shape. The common failure is no longer “the model wrote nothing.” It is silent expansion: extra modules, extra dependencies, extra comments that drift from the contract. Routing has to detect that shape before anyone debates style. Use these terms as they are defined here. Nearby words in vendor blogs do not override them. Walk the questions in order. Do not skip to a leaf because the diff “looks small.” Step 1 — Is there an oracle that already fails, or an oracle you can add in under fifteen minutes? Step 2 — Is surface area bounded? Bound means: requested paths only, or requested paths plus test files. A hard cap helps. A working default is “three production files and their tests.” Step 3 — Does the patch need network, secrets, or write access outside a temp directory? .env reads : go to Step 4 — Can an isolated process execute the oracle? The tree is deliberately biased toward discard and rewrite. Cheap generation makes “try it locally” the risky default, not the brave one. The script below does not prove safety. It only encodes the tree’s cheap heuristics so a human does not re-litigate them every morning. Label: proposal, unexecuted on your tree until you run it. bash /usr/bin/env python3 """classify patch.py — proposal heuristic, not a security scanner.""" from future import annotations import subprocess import sys from pathlib import Path ALLOWED PREFIXES = "src/", "lib/", "tests/", "test/" ARCH HINTS = "auth", "middleware", "migration", "dockerfile", "compose", ".github/" SECRET HINTS = "os.environ", "getenv ", "api key", "BEGIN ", ".env" MAX PROD FILES = 3 def git names diff range: str - list str : out = subprocess.check output "git", "diff", "--name-only", diff range , text=True return line.strip for line in out.splitlines if line.strip def patch text diff range: str - str: return subprocess.check output "git", "diff", diff range , text=True def classify diff range: str, requested: set str - str: names = git names diff range body = patch text diff range .lower prod = n for n in names if not Path n .parts 0 .startswith "test" extra = n for n in names if n not in requested and not n.startswith "test" if any h in n.lower for n in names for h in ARCH HINTS : return "LEAF D REWRITE architecture touch" if any h in body for h in SECRET HINTS : return "LEAF A DISCARD secret or env touch" if extra or len prod MAX PROD FILES: return "LEAF A DISCARD silent expansion" if not names: return "LEAF A DISCARD empty" if all n.startswith ALLOWED PREFIXES for n in names and len prod <= 2: return "LEAF C LOCAL allowlist" return "LEAF B QUARANTINE" if name == " main ": if len sys.argv < 3: print "usage: classify patch.py <diff-range <requested-file more files " sys.exit 2 decision = classify sys.argv 1 , set sys.argv 2: print decision Run it against a generated branch, not against main : git fetch origin git checkout -B agent/try-1 origin/agent/try-1 python3 classify patch.py main...HEAD src/billing/quote.py tests/test quote.py The printed leaf is a starting label. Override it when you have information the script cannot see, such as a compliance boundary or a frozen schema. Scene. Prompt: “Add a quote total items helper.” Diff: quote.py , logger.py , utils/retry.py , and a new requirements pin on a metrics SDK. Why this leaf. Surface area exploded. The extra files are not tests. The metrics SDK implies network. Step 2 and Step 3 both fail. Worked action. Task: add quote total items in src/billing/quote.py only. Do not create files. Do not edit requirements or logging. Oracle: tests/test quote.py::test quote total cents must pass. If the oracle needs a test change, edit that test file only. What “done” looks like. A new diff with one production file, or a decision to write the helper by hand because the review budget is already spent. Scene. Prompt: “Parse CSV invoices in src/invoices/parse.py .” Diff: that file plus tests/test parse.py . No auth, no Docker, no env reads. You do not want that parser executing against files in your home directory. Why this leaf. Bounded surface, no secret touch, and an oracle exists. Isolation is the remaining requirement. Worked action. Copy the branch into a throwaway directory or machine. Feed only fixture files. Run the oracle. Throw the machine state away. proposal workflow — run on a disposable host, not on a laptop with SSH keys loaded mkdir -p /tmp/quarantine && cd /tmp/quarantine git clone --depth 1 --branch agent/try-1 /path/to/local/mirror invoices cd invoices python -m venv .venv && . .venv/bin/activate pip install -e '. test ' pytest tests/test parse.py -q --fixtures-per-test If the oracle needs a CSV, mount a fixture directory that contains no customer data. If the model added requests.get , the run still belongs on Leaf A, even if pytest is green: the classifier should have caught it, and the quarantine host should have no egress if you can help it. A quarantine host can be a local container. It can also be a spare server that never sees production credentials. MonkeyCode is relevant on this leaf only: it currently offers free model access and a free server option, which is one way to keep generation and first-oracle runs off your workstation. Disclosure: This article was prepared as part of MonkeyCode's product outreach. Treat both the models and the server as capacity that can change; verify current terms on the project before you plan around them. Do not place secrets, customer fixtures, or deploy keys on a shared free server. What “done” looks like. Oracle green on the isolated copy, plus a human skim of the two files before allowlist apply onto a real branch. Scene. Prompt: “Extract cents amount: str - int .” Diff: src/money.py and tests/test money.py . Pure functions. No I/O. Classifier prints LEAF C LOCAL allowlist . Why this leaf. Isolation would still be nicer, but the blast radius is a two-file pure change and the oracle is local unit tests you already trust. Worked action. git checkout main git checkout agent/try-1 -- src/money.py tests/test money.py git diff --cached --stat pytest tests/test money.py -q Stop if --stat shows any other path. Stop if the test file grew assertions about logging, time, or HTTP. Those are expansion in disguise. What “done” looks like. Two allowlisted files, oracle green, no other staged paths. Scene. Prompt: “Make login less flaky.” Diff: auth/middleware.py , a session store swap, and a Dockerfile change to add Redis. Why this leaf. Architecture-touching. Step 3’s “real network need” is present, and there is no dedicated staging path in the scene. Quarantine cannot validate session semantics with unit tests alone. Local apply on a developer laptop is how flaky login becomes an outage. Worked action. proposal: freeze the contract before any generated middleware lands def test login failure body stable client : res = client.post "/login", json={"user": "x", "password": "bad"} assert res.status code == 401 assert set res.json .keys == {"error", "code"} What “done” looks like. A smaller patch that re-enters Leaf B or C, or a human-written change. Not a fourteen-file “login fix.” The tree does not detect vulnerabilities, license issues, or subtle numeric drift. The classifier is string heuristics. It will miss a polished secret read and it will over-flag a comment that mentions .env . Quarantine is not staging. A green oracle on a throwaway host does not mean production behavior. Free model access and a free server do not add an oracle; they only move generation and first execution off your laptop. If you have no contract tests, you do not have Leaf B. You have a remote place to watch the same untested code fail. Do not use this approach when the repo holds regulated data, when the agent needs production-like secrets, or when the change is a public contract you cannot freeze in a test. Do not use it as a substitute for architecture review on auth, payments, or migrations. Teams that cannot add a fifteen-minute oracle should not scale cheap generation; they should shrink the task. The routing bias toward discard will feel slow compared with “accept all files.” That slowness is the point. When patches are cheap, the scarce resource is attention. Spend it on oracles and surface-area caps, not on reading surprise Dockerfiles at 9:12 a.m. If Leaf B is the leaf you keep landing on, verify whether an isolated server is actually isolated, then run the oracle there. MonkeyCode’s free model access and free server option are one place to try that isolation; confirm current availability before you schedule work against them.