{"slug": "oracle-first-routing-ai-generated-diffs-with-a-glossary-and-four-leaves", "title": "Oracle First: Routing AI-Generated Diffs With a Glossary and Four Leaves", "summary": "An engineer proposes a routing framework for AI-generated code diffs, introducing a glossary and a decision tree to classify patches as keep, quarantine, or rewrite. The approach includes a heuristic classifier script that flags architecture touches, secret hints, and silent expansion, aiming to reduce the cost of reviewing cheaply generated code.", "body_md": "Consider this scene. It is a composite, not a personal war story.\n\nAn agent ran overnight on a leftover prompt. Morning `git status`\n\nshowed fourteen files. Two of them implemented the requested endpoint. The rest were a new logger, a renamed helper, a rewritten Dockerfile, and a README that now contradicted the tests. Generation cost was close to zero. The next four hours were not.\n\nThat gap is the actual product problem. When a patch is cheap to produce, the expensive work is routing: keep, quarantine, or rewrite. Skip the routing and the cheap code becomes expensive debt with extra files attached.\n\nThis article is a glossary, a routing tree, and a worked example at each leaf. The artifact is a small classifier plus a quarantine command sequence. Treat the code as a proposal unless you run it on your own repo.\n\nFree-model loops optimize for “a diff appeared.” Reviewers optimize for “this diff is safe to merge.” Those are different objective functions. A green unit test on a helper you did not ask for is not evidence that the architecture still holds.\n\nCheap generation also changes failure shape. The common failure is no longer “the model wrote nothing.” It is silent expansion: extra modules, extra dependencies, extra comments that drift from the contract. Routing has to detect that shape before anyone debates style.\n\nUse these terms as they are defined here. Nearby words in vendor blogs do not override them.\n\nWalk the questions in order. Do not skip to a leaf because the diff “looks small.”\n\n**Step 1 — Is there an oracle that already fails, or an oracle you can add in under fifteen minutes?**\n\n**Step 2 — Is surface area bounded?**\n\nBound means: requested paths only, or requested paths plus test files. A hard cap helps. A working default is “three production files and their tests.”\n\n**Step 3 — Does the patch need network, secrets, or write access outside a temp directory?**\n\n`.env`\n\nreads): go to **Step 4 — Can an isolated process execute the oracle?**\n\nThe tree is deliberately biased toward discard and rewrite. Cheap generation makes “try it locally” the risky default, not the brave one.\n\nThe script below does not prove safety. It only encodes the tree’s cheap heuristics so a human does not re-litigate them every morning. Label: proposal, unexecuted on your tree until you run it.\n\n``` bash\n#!/usr/bin/env python3\n\"\"\"classify_patch.py — proposal heuristic, not a security scanner.\"\"\"\nfrom __future__ import annotations\n\nimport subprocess\nimport sys\nfrom pathlib import Path\n\nALLOWED_PREFIXES = (\"src/\", \"lib/\", \"tests/\", \"test/\")\nARCH_HINTS = (\"auth\", \"middleware\", \"migration\", \"dockerfile\", \"compose\", \".github/\")\nSECRET_HINTS = (\"os.environ\", \"getenv(\", \"api_key\", \"BEGIN \", \".env\")\nMAX_PROD_FILES = 3\n\ndef git_names(diff_range: str) -> list[str]:\n    out = subprocess.check_output(\n        [\"git\", \"diff\", \"--name-only\", diff_range], text=True\n    )\n    return [line.strip() for line in out.splitlines() if line.strip()]\n\ndef patch_text(diff_range: str) -> str:\n    return subprocess.check_output([\"git\", \"diff\", diff_range], text=True)\n\ndef classify(diff_range: str, requested: set[str]) -> str:\n    names = git_names(diff_range)\n    body = patch_text(diff_range).lower()\n    prod = [n for n in names if not Path(n).parts[0].startswith(\"test\")]\n    extra = [n for n in names if n not in requested and not n.startswith(\"test\")]\n\n    if any(h in n.lower() for n in names for h in ARCH_HINTS):\n        return \"LEAF_D_REWRITE_architecture_touch\"\n    if any(h in body for h in SECRET_HINTS):\n        return \"LEAF_A_DISCARD_secret_or_env_touch\"\n    if extra or len(prod) > MAX_PROD_FILES:\n        return \"LEAF_A_DISCARD_silent_expansion\"\n    if not names:\n        return \"LEAF_A_DISCARD_empty\"\n    if all(n.startswith(ALLOWED_PREFIXES) for n in names) and len(prod) <= 2:\n        return \"LEAF_C_LOCAL_allowlist\"\n    return \"LEAF_B_QUARANTINE\"\n\nif __name__ == \"__main__\":\n    if len(sys.argv) < 3:\n        print(\"usage: classify_patch.py <diff-range> <requested-file> [more files]\")\n        sys.exit(2)\n    decision = classify(sys.argv[1], set(sys.argv[2:]))\n    print(decision)\n```\n\nRun it against a generated branch, not against `main`\n\n:\n\n```\ngit fetch origin\ngit checkout -B agent/try-1 origin/agent/try-1\npython3 classify_patch.py main...HEAD src/billing/quote.py tests/test_quote.py\n```\n\nThe printed leaf is a starting label. Override it when you have information the script cannot see, such as a compliance boundary or a frozen schema.\n\n**Scene.** Prompt: “Add a `quote_total(items)`\n\nhelper.” Diff: `quote.py`\n\n, `logger.py`\n\n, `utils/retry.py`\n\n, and a new `requirements`\n\npin on a metrics SDK.\n\n**Why this leaf.** Surface area exploded. The extra files are not tests. The metrics SDK implies network. Step 2 and Step 3 both fail.\n\n**Worked action.**\n\n```\nTask: add quote_total(items) in src/billing/quote.py only.\nDo not create files. Do not edit requirements or logging.\nOracle: tests/test_quote.py::test_quote_total_cents must pass.\nIf the oracle needs a test change, edit that test file only.\n```\n\n**What “done” looks like.** A new diff with one production file, or a decision to write the helper by hand because the review budget is already spent.\n\n**Scene.** Prompt: “Parse CSV invoices in `src/invoices/parse.py`\n\n.” Diff: that file plus `tests/test_parse.py`\n\n. No auth, no Docker, no env reads. You do not want that parser executing against files in your home directory.\n\n**Why this leaf.** Bounded surface, no secret touch, and an oracle exists. Isolation is the remaining requirement.\n\n**Worked action.** Copy the branch into a throwaway directory or machine. Feed only fixture files. Run the oracle. Throw the machine state away.\n\n```\n# proposal workflow — run on a disposable host, not on a laptop with SSH keys loaded\nmkdir -p /tmp/quarantine && cd /tmp/quarantine\ngit clone --depth 1 --branch agent/try-1 /path/to/local/mirror invoices\ncd invoices\npython -m venv .venv && . .venv/bin/activate\npip install -e '.[test]'\npytest tests/test_parse.py -q --fixtures-per-test\n```\n\nIf the oracle needs a CSV, mount a fixture directory that contains no customer data. If the model added `requests.get`\n\n, the run still belongs on Leaf A, even if pytest is green: the classifier should have caught it, and the quarantine host should have no egress if you can help it.\n\nA quarantine host can be a local container. It can also be a spare server that never sees production credentials. MonkeyCode is relevant on this leaf only: it currently offers free model access and a free server option, which is one way to keep generation and first-oracle runs off your workstation. Disclosure: This article was prepared as part of MonkeyCode's product outreach. Treat both the models and the server as capacity that can change; verify current terms on the project before you plan around them. Do not place secrets, customer fixtures, or deploy keys on a shared free server.\n\n**What “done” looks like.** Oracle green on the isolated copy, plus a human skim of the two files before allowlist apply onto a real branch.\n\n**Scene.** Prompt: “Extract `cents(amount: str) -> int`\n\n.” Diff: `src/money.py`\n\nand `tests/test_money.py`\n\n. Pure functions. No I/O. Classifier prints `LEAF_C_LOCAL_allowlist`\n\n.\n\n**Why this leaf.** Isolation would still be nicer, but the blast radius is a two-file pure change and the oracle is local unit tests you already trust.\n\n**Worked action.**\n\n```\ngit checkout main\ngit checkout agent/try-1 -- src/money.py tests/test_money.py\ngit diff --cached --stat\npytest tests/test_money.py -q\n```\n\nStop if `--stat`\n\nshows any other path. Stop if the test file grew assertions about logging, time, or HTTP. Those are expansion in disguise.\n\n**What “done” looks like.** Two allowlisted files, oracle green, no other staged paths.\n\n**Scene.** Prompt: “Make login less flaky.” Diff: `auth/middleware.py`\n\n, a session store swap, and a Dockerfile change to add Redis.\n\n**Why this leaf.** Architecture-touching. Step 3’s “real network need” is present, and there is no dedicated staging path in the scene. Quarantine cannot validate session semantics with unit tests alone. Local apply on a developer laptop is how flaky login becomes an outage.\n\n**Worked action.**\n\n```\n# proposal: freeze the contract before any generated middleware lands\ndef test_login_failure_body_stable(client):\n    res = client.post(\"/login\", json={\"user\": \"x\", \"password\": \"bad\"})\n    assert res.status_code == 401\n    assert set(res.json().keys()) == {\"error\", \"code\"}\n```\n\n**What “done” looks like.** A smaller patch that re-enters Leaf B or C, or a human-written change. Not a fourteen-file “login fix.”\n\nThe tree does not detect vulnerabilities, license issues, or subtle numeric drift. The classifier is string heuristics. It will miss a polished secret read and it will over-flag a comment that mentions `.env`\n\n.\n\nQuarantine is not staging. A green oracle on a throwaway host does not mean production behavior. Free model access and a free server do not add an oracle; they only move generation and first execution off your laptop. If you have no contract tests, you do not have Leaf B. You have a remote place to watch the same untested code fail.\n\nDo not use this approach when the repo holds regulated data, when the agent needs production-like secrets, or when the change is a public contract you cannot freeze in a test. Do not use it as a substitute for architecture review on auth, payments, or migrations. Teams that cannot add a fifteen-minute oracle should not scale cheap generation; they should shrink the task.\n\nThe routing bias toward discard will feel slow compared with “accept all files.” That slowness is the point. When patches are cheap, the scarce resource is attention. Spend it on oracles and surface-area caps, not on reading surprise Dockerfiles at 9:12 a.m.\n\nIf Leaf B is the leaf you keep landing on, verify whether an isolated server is actually isolated, then run the oracle there. MonkeyCode’s free model access and free server option are one place to try that isolation; confirm current availability before you schedule work against them.", "url": "https://wpnews.pro/news/oracle-first-routing-ai-generated-diffs-with-a-glossary-and-four-leaves", "canonical_source": "https://dev.to/devpy_9520/oracle-first-routing-ai-generated-diffs-with-a-glossary-and-four-leaves-ac5", "published_at": "2026-09-03 15:16:45+00:00", "updated_at": "2026-09-03 15:26:45.248796+00:00", "lang": "en", "topics": ["developer-tools", "ai-agents", "ai-tools"], "entities": [], "alternates": {"html": "https://wpnews.pro/news/oracle-first-routing-ai-generated-diffs-with-a-glossary-and-four-leaves", "markdown": "https://wpnews.pro/news/oracle-first-routing-ai-generated-diffs-with-a-glossary-and-four-leaves.md", "text": "https://wpnews.pro/news/oracle-first-routing-ai-generated-diffs-with-a-glossary-and-four-leaves.txt", "jsonld": "https://wpnews.pro/news/oracle-first-routing-ai-generated-diffs-with-a-glossary-and-four-leaves.jsonld"}}