{"slug": "validating-llm-code-edits-when-you-can-t-run-the-code", "title": "Validating LLM code edits when you can't run the code", "summary": "A developer describes a five-layer validation stack for LLM-generated code edits in repositories where the code cannot be executed. The stack constrains file access, anchors edits to actual file bytes, runs static checks, validates unparseable formats comparatively, and uses an adversarial second model. Each layer fails with a distinct status to avoid retry loops, ensuring provably non-broken changes before human review.", "body_md": "# Validating LLM Code Edits When You Can't Run the Code\n\nAn agent that writes code into repositories it cannot execute gets none of the usual safety nets: no build, no tests, no typecheck. What remains is a five-layer stack: constrain which files are touchable, anchor every edit to real file bytes instead of the model's quotation of them, run static checks that only reject provable breakage, validate unparseable formats comparatively (did the edit make the file worse), and put an adversarial second model between the proposal and the write. Every layer fails into a distinct, honest status instead of a retry loop.\n\nThere is a class of system that has to write code into repositories it cannot execute. Ours is an agent that opens pull requests against other people's production websites. It runs in a serverless isolate with a wall-clock budget, reads the repo over the GitHub API, and at the moment it writes a change it has no checkout, no `node_modules`\n\n, no build, no test run, and no dev server. Whatever validation happens has to happen before the PR exists, using nothing but the bytes of the files, static analysis that runs in milliseconds, and more model calls.\n\nThe customer's own CI may run on the PR afterwards. That is not a safety net, it is someone else's inbox. A PR that fails CI on a syntax error costs the exact trust the whole system depends on. The goal is a change that is provably not-broken before any human sees it.\n\nAfter a year of production runs, the validation stack has settled into five layers. Each one is cheap. Each one rejects only what it can prove is broken. And they are ordered so that a failure at any point leaves nothing behind in the target repo.\n\n## What \"can't run the code\" actually rules out\n\nIt is worth being precise, because each unavailable tool kills a familiar guard:\n\n**No** Arbitrary package managers, private registries, native dependencies, and minutes of wall clock in an environment budgeted in seconds. This rules out running anything.`npm install`\n\n.**No test suite.** Even if one exists (you don't know), you can't run it.**No typecheck.**`tsc`\n\nneeds the full dependency graph, which needs the install.**No rendering.** You cannot boot the app and look at it. (You can screenshot the live production site, which turns out to matter later.)\n\nSo the usual ladder (types, tests, CI, staging) is gone at write time. What's left is the file itself, before and after the edit.\n\n## Layer 1: constrain the blast radius before validating anything\n\nThe first guard has nothing to do with code quality. A model choosing which file to edit is a model that can choose `.github/workflows/deploy.yml`\n\n, and a workflow edit is a secret-exfiltration primitive: CI runs with credentials the agent was never given. So before any content check, the chosen path runs against a denylist:\n\n``` js\nconst FORBIDDEN_EDIT_PATHS: RegExp[] = [\n /^\\.github\\//i, // CI can exfiltrate secrets\n /(^|\\/)\\.env(\\.|$)/i, // .env, .env.local, .env.production\n /(^|\\/)package(-lock)?\\.json$/i, // dependency manifests\n /(^|\\/)vercel\\.json$/i, // deploy config\n /\\.pem$|\\.key$|\\.p12$|\\.pfx$/i, // private keys\n /(^|\\/)supabase\\/functions\\//i, // the agent itself (no self-modification)\n // ...framework configs, lockfiles, Dockerfiles, IaC, migrations\n]\n```\n\nThen the inverse gate: an allowlist of file extensions the pipeline can actually verify. The syntax validator returns `ok: true`\n\nfor types it cannot parse (`.vue`\n\n, `.svelte`\n\n), because a validator that blocks everything it doesn't understand blocks too much. That pass-through would be a hole, so the caller closes it: any extension outside the verifiable set is refused outright, with an error that says exactly why (\"refusing to open an unverified PR\"). A permissive validator plus a strict extension gate is fail-closed. A permissive validator alone is not.\n\n## Layer 2: anchor edits to real bytes, not the model's memory of them\n\nThe model emits edits as find/replace pairs. The dominant failure mode is not malice or hallucinated APIs, it is **transcription drift**: the `find`\n\nstring is a near-copy of the real code with slightly different whitespace, a straightened quote, or reordered attributes. Applied naively, that either fails or, worse, matches the wrong thing.\n\nThe guard works in two steps. First, try the exact substring. If that fails, build a whitespace-normalized version of the file alongside an index map, so every character of the normalized text points back at its position in the original:\n\n```\n// Collapse whitespace runs to a single space AND record, per\n// normalized char, the original index it came from, so a normalized\n// match maps back to exact original bytes.\nfunction buildNormalized(content: string): { norm: string; map: number[] }\n```\n\nOn a unique match, the replacement is spliced into **the actual bytes at that anchor**, never into the model's quotation of them. The model's `find`\n\nis only ever used to locate; the file's own content is what survives around the edit.\n\nThe two failure cases get different names, and that matters operationally:\n\n**Zero matches** becomes`find_mismatch`\n\n, and the error carries the closest-scoring lines from the real file, so the failure is diagnosable from the record alone.**Multiple matches** becomes`find_ambiguous`\n\n, with a snippet of each candidate site.\n\nThese are distinct statuses in the database, never a generic `failed`\n\n. A rising mismatch rate means the model is paraphrasing what it quotes; a rising ambiguity rate means its anchors are too short. You cannot see either trend through a single undifferentiated failure bucket.\n\nThere is exactly one self-heal: on a mismatch, the model is re-asked to re-anchor its `find`\n\nagainst the file's real content. The `replace`\n\nis never altered, so the repair can fix location but never semantics. If it misses twice, the run ends in `find_mismatch`\n\n, honestly.\n\n## Layer 3: static checks that reject only provable breakage\n\nFor file types with a real parser, this layer is boring and good: Babel with `errorRecovery: false`\n\nfor the JS/TS family, `JSON.parse`\n\nfor JSON. A dropped brace never reaches a commit.\n\nLiquid templates (Shopify themes) are where it gets interesting, because there is no strict parser you can borrow, and theme files legitimately contain fragments that look broken. The delimiter check is **asymmetric by design**: it flags an opening `{{`\n\nor `{%`\n\nthat never closes, and deliberately never flags a stray `}}`\n\nor `%}`\n\n:\n\n```\n// Asymmetry by design: we flag an OPENING ({{ or {%) that never\n// closes, but NOT a stray CLOSING, because bare braces occur\n// constantly in JS/CSS inside theme files. Catching dropped-close\n// is the high-value common case; flagging stray-close would\n// false-reject valid themes wholesale.\n```\n\nA dropped closing delimiter is the high-frequency LLM failure. A stray closing brace is what inline `