Validating LLM code edits when you can't run the code A developer describes a five-layer validation stack for LLM-generated code edits in repositories where the code cannot be executed. The stack constrains file access, anchors edits to actual file bytes, runs static checks, validates unparseable formats comparatively, and uses an adversarial second model. Each layer fails with a distinct status to avoid retry loops, ensuring provably non-broken changes before human review. Validating LLM Code Edits When You Can't Run the Code An agent that writes code into repositories it cannot execute gets none of the usual safety nets: no build, no tests, no typecheck. What remains is a five-layer stack: constrain which files are touchable, anchor every edit to real file bytes instead of the model's quotation of them, run static checks that only reject provable breakage, validate unparseable formats comparatively did the edit make the file worse , and put an adversarial second model between the proposal and the write. Every layer fails into a distinct, honest status instead of a retry loop. There is a class of system that has to write code into repositories it cannot execute. Ours is an agent that opens pull requests against other people's production websites. It runs in a serverless isolate with a wall-clock budget, reads the repo over the GitHub API, and at the moment it writes a change it has no checkout, no node modules , no build, no test run, and no dev server. Whatever validation happens has to happen before the PR exists, using nothing but the bytes of the files, static analysis that runs in milliseconds, and more model calls. The customer's own CI may run on the PR afterwards. That is not a safety net, it is someone else's inbox. A PR that fails CI on a syntax error costs the exact trust the whole system depends on. The goal is a change that is provably not-broken before any human sees it. After a year of production runs, the validation stack has settled into five layers. Each one is cheap. Each one rejects only what it can prove is broken. And they are ordered so that a failure at any point leaves nothing behind in the target repo. What "can't run the code" actually rules out It is worth being precise, because each unavailable tool kills a familiar guard: No Arbitrary package managers, private registries, native dependencies, and minutes of wall clock in an environment budgeted in seconds. This rules out running anything. npm install . No test suite. Even if one exists you don't know , you can't run it. No typecheck. tsc needs the full dependency graph, which needs the install. No rendering. You cannot boot the app and look at it. You can screenshot the live production site, which turns out to matter later. So the usual ladder types, tests, CI, staging is gone at write time. What's left is the file itself, before and after the edit. Layer 1: constrain the blast radius before validating anything The first guard has nothing to do with code quality. A model choosing which file to edit is a model that can choose .github/workflows/deploy.yml , and a workflow edit is a secret-exfiltration primitive: CI runs with credentials the agent was never given. So before any content check, the chosen path runs against a denylist: js const FORBIDDEN EDIT PATHS: RegExp = /^\.github\//i, // CI can exfiltrate secrets / ^|\/ \.env \.|$ /i, // .env, .env.local, .env.production / ^|\/ package -lock ?\.json$/i, // dependency manifests / ^|\/ vercel\.json$/i, // deploy config /\.pem$|\.key$|\.p12$|\.pfx$/i, // private keys / ^|\/ supabase\/functions\//i, // the agent itself no self-modification // ...framework configs, lockfiles, Dockerfiles, IaC, migrations Then the inverse gate: an allowlist of file extensions the pipeline can actually verify. The syntax validator returns ok: true for types it cannot parse .vue , .svelte , because a validator that blocks everything it doesn't understand blocks too much. That pass-through would be a hole, so the caller closes it: any extension outside the verifiable set is refused outright, with an error that says exactly why "refusing to open an unverified PR" . A permissive validator plus a strict extension gate is fail-closed. A permissive validator alone is not. Layer 2: anchor edits to real bytes, not the model's memory of them The model emits edits as find/replace pairs. The dominant failure mode is not malice or hallucinated APIs, it is transcription drift : the find string is a near-copy of the real code with slightly different whitespace, a straightened quote, or reordered attributes. Applied naively, that either fails or, worse, matches the wrong thing. The guard works in two steps. First, try the exact substring. If that fails, build a whitespace-normalized version of the file alongside an index map, so every character of the normalized text points back at its position in the original: // Collapse whitespace runs to a single space AND record, per // normalized char, the original index it came from, so a normalized // match maps back to exact original bytes. function buildNormalized content: string : { norm: string; map: number } On a unique match, the replacement is spliced into the actual bytes at that anchor , never into the model's quotation of them. The model's find is only ever used to locate; the file's own content is what survives around the edit. The two failure cases get different names, and that matters operationally: Zero matches becomes find mismatch , and the error carries the closest-scoring lines from the real file, so the failure is diagnosable from the record alone. Multiple matches becomes find ambiguous , with a snippet of each candidate site. These are distinct statuses in the database, never a generic failed . A rising mismatch rate means the model is paraphrasing what it quotes; a rising ambiguity rate means its anchors are too short. You cannot see either trend through a single undifferentiated failure bucket. There is exactly one self-heal: on a mismatch, the model is re-asked to re-anchor its find against the file's real content. The replace is never altered, so the repair can fix location but never semantics. If it misses twice, the run ends in find mismatch , honestly. Layer 3: static checks that reject only provable breakage For file types with a real parser, this layer is boring and good: Babel with errorRecovery: false for the JS/TS family, JSON.parse for JSON. A dropped brace never reaches a commit. Liquid templates Shopify themes are where it gets interesting, because there is no strict parser you can borrow, and theme files legitimately contain fragments that look broken. The delimiter check is asymmetric by design : it flags an opening {{ or {% that never closes, and deliberately never flags a stray }} or %} : // Asymmetry by design: we flag an OPENING {{ or {% that never // closes, but NOT a stray CLOSING, because bare braces occur // constantly in JS/CSS inside theme files. Catching dropped-close // is the high-value common case; flagging stray-close would // false-reject valid themes wholesale. A dropped closing delimiter is the high-frequency LLM failure. A stray closing brace is what inline