Our verifier was buggier than the code it verified

Refactyl, a codebase migration tool, shipped a bug in its verification pipeline where a false fail rejected valid Vue 3 output during a live migration. The bug was hidden by the pipeline's safety mechanism, which silently downgraded correct conversions instead of shipping broken code. The issue was traced to an optimization that reused a pre-parsed AST in the compiler gate, which failed for certain v-if and slot structures.

The most dangerous bug in a verification pipeline isn't a false pass. It's a false fail. We know because we shipped one. Refactyl migrates codebases — Vue 2 to Vue 3, JS to TS, Pages Router to App Router — and the whole pitch is that a compiler gate decides what ships, not vibes. This is the story of the gate itself being wrong: valid Vue 3 output, rejected by our own verifier, on a live migration. The conversion was correct. The code checking it was not. If you build anything with a validation step — CI, linters, schema checks, LLM output filters — the failure taxonomy here applies to you. Every converted Vue single-file component in our pipeline must pass @vue/compiler-sfc before it ships. Not a regex. Not a diff heuristic. The actual compiler Vue itself uses. During a live migration, converted components started failing that gate with this: Codegen node is missing for element/if/for node. Which is a strange error to get, because the components were fine. Paste the same output into a fresh Vue 3 project and it compiles. The migration output was valid; our verifier said it wasn't. Here's the part that matters. Our gate is designed to fail safely. If a converted file doesn't pass the compiler, we don't ship it broken — we keep the original file, flag it, and move on. Flagged, never silently broken. That design is correct. It is also exactly why this bug was invisible. When the gate falsely rejected a good conversion, nothing crashed. No exception reached a log that anyone treated as an error. No user saw broken code. The pipeline did precisely what it was built to do: quietly downgraded a correct conversion to "keep the original, flag it." The metrics that suffered — conversion rate down, flag rate up — look like the tool being appropriately cautious, not like a defect. A false pass announces itself eventually. Someone's build breaks, someone files an issue, you trace it back. A false fail is absorbed by the safety mechanism. The fallback path you built to make failure harmless is the same path that hides the bug. The gate lived in engines/vue2-vue3/sfc-gate.ts:59-63 . Simplified, the broken version looked like this: js import { parse, compileTemplate } from '@vue/compiler-sfc'; const { descriptor, errors } = parse source ; if errors.length return reject errors ; // fast path: we already parsed the SFC, reuse the AST const result = compileTemplate { source: descriptor.template.content, ast: descriptor.template.ast, // <-- the bug id, filename, } ; The reasoning felt sound. parse already produced a template AST. compileTemplate accepts one. Re-parsing the same source twice is wasted work, so hand the AST over. Except compileTemplate makes assumptions about the ASTs it receives — assumptions the descriptor AST from parse doesn't meet for certain v-if and slot structures. On those shapes, the code generator reaches for transform metadata that was never attached and throws Codegen node is missing . Verified live: components with those structures — valid Vue 3, accepted by the compiler when fed as source — failed our gate every time. The verifier's fast path was buggier than any conversion it rejected. The rewritten gate does two compilations, and both start from source text. It never verifies from a pre-parsed AST. // stage 1: parse from source — structural validity const { descriptor, errors } = parse source ; if errors.length return reject errors ; // stage 2: compileTemplate from source — directive-level errors // that parse tolerates const result = compileTemplate { source: descriptor.template.content, // string, not AST id, filename, } ; if result.errors.length return reject result.errors ; Why two stages instead of just compileTemplate ? Because the stages catch different failure classes. parse catches structural problems — malformed templates, unclosed tags. But parse tolerates directive errors that only surface at codegen: a v-else with no preceding v-if parses cleanly and fails compilation. You need both passes to cover both classes. And why from source, both times? Because source text is the one representation with no hidden coupling. An AST carries assumptions — which parser produced it, with which options, with which transforms already applied. Hand it across an API boundary and you're betting the receiver shares every assumption. The compiler your users run consumes source. Your gate should consume exactly what the real compiler consumes, or it is verifying something subtly different from what ships. The fast path existed for a real reason — re-parsing costs time. We pay that cost now. A verifier that's fast and wrong is worth less than no verifier, because it converts good work into rejected work and calls it safety. Every gate has two failure modes, and their costs are asymmetric in a way that shapes where your bugs hide. A false pass ships broken code. Expensive, visible, and self-correcting: the damage lands on a user, the user complains, you fix it. Painful but loud. A false fail silently loses good work. Nothing breaks. Nothing errors. Your product just gets quietly worse — lower conversion rates, more flags, more "we couldn't convert this safely" on files that could have converted fine. Nobody files a bug against behavior that looks like designed caution. We have scars on both sides of this line, in the same codebase. The under-catching side: our Next.js Pages-to-App-Router engine has a lint engines/nextjs-pages-app/transformer.ts:66-108 that exists only because parsers miss things. Model output that keeps a getServerSideProps export "for reference," imports next/router in App Router code, or writes a page whose default export destructures { post } — when App Router only ever passes { params, searchParams } — parses cleanly and then fails next build or crashes at prerender. Syntax-level gates under-reject. Each rule in that lint is a fossil of a real failure that got past parsing. The designed-fallback side: when a generated Next.js root layout fails its gate, we don't ship nothing — a deterministic buildMinimalLayout ships instead transformer.ts:220-265 , because a missing root layout is a build error and degraded beats unbuildable. That's what a false-fail response should look like: explicit, deterministic, and producing output the compiler accepts. The SFC gate was the third case: over-rejecting, with the fallback masking it. Same pipeline, three lessons. Three rules came out of this. None are Vue-specific. Treat the verifier as production code. Rejection tests verify the gate can say no — invalid SFCs it must reject. Acceptance tests verify it can say yes on known-good input across the shapes that matter: v-if chains, named slots, scoped slots. The acceptance direction is the one that catches false fails, and it's the coverage we're building from this incident's corpus. If your gate's test suite is mostly bad inputs, half its behavior is untested. Instrument the fallback path like an error path. "Keep original and flag" is a designed outcome, and the instinct is to log designed outcomes as information. Wrong altitude. A flag rate that drifts upward is an alert, not a statistic. Any path whose job is absorbing failure deserves the same observability as a crash, because it absorbs your bugs too. Verify from the artifact you ship. Not from an intermediate representation, however convenient. The distance between "what the gate checked" and "what the user's compiler sees" is exactly the space bugs like this live in. Naming what this doesn't prove is part of the method. The fix removes one class of false fail — AST reuse. It does not make the gate bug-free, and we won't claim that; a verifier is code, and code has bugs. Acceptance coverage is only as good as the corpus of valid shapes we thought to include; a structure we haven't covered could still be falsely rejected. When that happens, the file ships unchanged and flagged — the designed behavior, and still a real cost to the user who wanted it converted. And the gate checks compilability, not runtime behavior: @vue/compiler-sfc accepting a component doesn't prove the component does what the Vue 2 original did. The gate decides what ships. This bug is why we stopped assuming the gate itself is above suspicion. "Flagged, never silently broken" is only a promise if the flagger is right. For a while, ours wasn't, and the safest-looking path in the pipeline is where the bug hid. If you're building verification into anything — a migration pipeline, a CI gate, an LLM output filter — audit the direction nobody watches. Your false passes will find you. Your false fails, you have to go find. Our migration methodology and the numbers behind it are at refactyl.com/benchmark https://refactyl.com/benchmark . If you want to see the gate work on your own code, the playground is at refactyl.com https://refactyl.com — start free migration.