{"slug": "our-verifier-was-buggier-than-the-code-it-verified", "title": "Our verifier was buggier than the code it verified", "summary": "Refactyl, a codebase migration tool, shipped a bug in its verification pipeline where a false fail rejected valid Vue 3 output during a live migration. The bug was hidden by the pipeline's safety mechanism, which silently downgraded correct conversions instead of shipping broken code. The issue was traced to an optimization that reused a pre-parsed AST in the compiler gate, which failed for certain v-if and slot structures.", "body_md": "The most dangerous bug in a verification pipeline isn't a false pass. It's a false fail.\n\nWe know because we shipped one.\n\nRefactyl migrates codebases — Vue 2 to Vue 3, JS to TS, Pages Router to App Router — and the whole pitch is that a compiler gate decides what ships, not vibes. This is the story of the gate itself being wrong: valid Vue 3 output, rejected by our own verifier, on a live migration. The conversion was correct. The code checking it was not.\n\nIf you build anything with a validation step — CI, linters, schema checks, LLM output filters — the failure taxonomy here applies to you.\n\nEvery converted Vue single-file component in our pipeline must pass `@vue/compiler-sfc`\n\nbefore it ships. Not a regex. Not a diff heuristic. The actual compiler Vue itself uses.\n\nDuring a live migration, converted components started failing that gate with this:\n\n```\nCodegen node is missing for element/if/for node.\n```\n\nWhich is a strange error to get, because the components were fine. Paste the same output into a fresh Vue 3 project and it compiles. The migration output was valid; our verifier said it wasn't.\n\nHere's the part that matters. Our gate is designed to fail safely. If a converted file doesn't pass the compiler, we don't ship it broken — we keep the original file, flag it, and move on. Flagged, never silently broken.\n\nThat design is correct. It is also exactly why this bug was invisible.\n\nWhen the gate falsely rejected a good conversion, nothing crashed. No exception reached a log that anyone treated as an error. No user saw broken code. The pipeline did precisely what it was built to do: quietly downgraded a correct conversion to \"keep the original, flag it.\" The metrics that suffered — conversion rate down, flag rate up — look like the tool being appropriately cautious, not like a defect.\n\nA false pass announces itself eventually. Someone's build breaks, someone files an issue, you trace it back. A false fail is absorbed by the safety mechanism. The fallback path you built to make failure harmless is the same path that hides the bug.\n\nThe gate lived in `engines/vue2-vue3/sfc-gate.ts:59-63`\n\n. Simplified, the broken version looked like this:\n\n``` js\nimport { parse, compileTemplate } from '@vue/compiler-sfc';\n\nconst { descriptor, errors } = parse(source);\nif (errors.length) return reject(errors);\n\n// fast path: we already parsed the SFC, reuse the AST\nconst result = compileTemplate({\n  source: descriptor.template.content,\n  ast: descriptor.template.ast,   // <-- the bug\n  id,\n  filename,\n});\n```\n\nThe reasoning felt sound. `parse()`\n\nalready produced a template AST. `compileTemplate()`\n\naccepts one. Re-parsing the same source twice is wasted work, so hand the AST over.\n\nExcept `compileTemplate()`\n\nmakes assumptions about the ASTs it receives — assumptions the descriptor AST from `parse()`\n\ndoesn't meet for certain `v-if`\n\nand slot structures. On those shapes, the code generator reaches for transform metadata that was never attached and throws `Codegen node is missing`\n\n. Verified live: components with those structures — valid Vue 3, accepted by the compiler when fed as source — failed our gate every time.\n\nThe verifier's fast path was buggier than any conversion it rejected.\n\nThe rewritten gate does two compilations, and both start from source text. It never verifies from a pre-parsed AST.\n\n```\n// stage 1: parse() from source — structural validity\nconst { descriptor, errors } = parse(source);\nif (errors.length) return reject(errors);\n\n// stage 2: compileTemplate() from source — directive-level errors\n// that parse() tolerates\nconst result = compileTemplate({\n  source: descriptor.template.content,  // string, not AST\n  id,\n  filename,\n});\nif (result.errors.length) return reject(result.errors);\n```\n\nWhy two stages instead of just `compileTemplate()`\n\n? Because the stages catch different failure classes. `parse()`\n\ncatches structural problems — malformed templates, unclosed tags. But `parse()`\n\ntolerates directive errors that only surface at codegen: a `v-else`\n\nwith no preceding `v-if`\n\nparses cleanly and fails compilation. You need both passes to cover both classes.\n\nAnd why from source, both times? Because source text is the one representation with no hidden coupling. An AST carries assumptions — which parser produced it, with which options, with which transforms already applied. Hand it across an API boundary and you're betting the receiver shares every assumption. The compiler your users run consumes source. Your gate should consume exactly what the real compiler consumes, or it is verifying something subtly different from what ships.\n\nThe fast path existed for a real reason — re-parsing costs time. We pay that cost now. A verifier that's fast and wrong is worth less than no verifier, because it converts good work into rejected work and calls it safety.\n\nEvery gate has two failure modes, and their costs are asymmetric in a way that shapes where your bugs hide.\n\n**A false pass ships broken code.** Expensive, visible, and self-correcting: the damage lands on a user, the user complains, you fix it. Painful but loud.\n\n**A false fail silently loses good work.** Nothing breaks. Nothing errors. Your product just gets quietly worse — lower conversion rates, more flags, more \"we couldn't convert this safely\" on files that could have converted fine. Nobody files a bug against behavior that looks like designed caution.\n\nWe have scars on both sides of this line, in the same codebase.\n\nThe under-catching side: our Next.js Pages-to-App-Router engine has a lint (`engines/nextjs-pages-app/transformer.ts:66-108`\n\n) that exists only because parsers miss things. Model output that keeps a `getServerSideProps`\n\nexport \"for reference,\" imports `next/router`\n\nin App Router code, or writes a page whose default export destructures `{ post }`\n\n— when App Router only ever passes `{ params, searchParams }`\n\n— parses cleanly and then fails `next build`\n\nor crashes at prerender. Syntax-level gates under-reject. Each rule in that lint is a fossil of a real failure that got past parsing.\n\nThe designed-fallback side: when a generated Next.js root layout fails its gate, we don't ship nothing — a deterministic `buildMinimalLayout()`\n\nships instead (`transformer.ts:220-265`\n\n), because a missing root layout is a build error and degraded beats unbuildable. That's what a false-fail response should look like: explicit, deterministic, and producing output the compiler accepts.\n\nThe SFC gate was the third case: over-rejecting, with the fallback masking it. Same pipeline, three lessons.\n\nThree rules came out of this. None are Vue-specific.\n\n**Treat the verifier as production code.** Rejection tests verify the gate can say no — invalid SFCs it must reject. Acceptance tests verify it can say yes on known-good input across the shapes that matter: `v-if`\n\nchains, named slots, scoped slots. The acceptance direction is the one that catches false fails, and it's the coverage we're building from this incident's corpus. If your gate's test suite is mostly bad inputs, half its behavior is untested.\n\n**Instrument the fallback path like an error path.** \"Keep original and flag\" is a designed outcome, and the instinct is to log designed outcomes as information. Wrong altitude. A flag rate that drifts upward is an alert, not a statistic. Any path whose job is absorbing failure deserves the same observability as a crash, because it absorbs your bugs too.\n\n**Verify from the artifact you ship.** Not from an intermediate representation, however convenient. The distance between \"what the gate checked\" and \"what the user's compiler sees\" is exactly the space bugs like this live in.\n\nNaming what this doesn't prove is part of the method.\n\nThe fix removes one class of false fail — AST reuse. It does not make the gate bug-free, and we won't claim that; a verifier is code, and code has bugs. Acceptance coverage is only as good as the corpus of valid shapes we thought to include; a structure we haven't covered could still be falsely rejected. When that happens, the file ships unchanged and flagged — the designed behavior, and still a real cost to the user who wanted it converted. And the gate checks compilability, not runtime behavior: `@vue/compiler-sfc`\n\naccepting a component doesn't prove the component does what the Vue 2 original did.\n\nThe gate decides what ships. This bug is why we stopped assuming the gate itself is above suspicion.\n\n\"Flagged, never silently broken\" is only a promise if the flagger is right. For a while, ours wasn't, and the safest-looking path in the pipeline is where the bug hid.\n\nIf you're building verification into anything — a migration pipeline, a CI gate, an LLM output filter — audit the direction nobody watches. Your false passes will find you. Your false fails, you have to go find.\n\nOur migration methodology and the numbers behind it are at [refactyl.com/benchmark](https://refactyl.com/benchmark). If you want to see the gate work on your own code, the playground is at [refactyl.com](https://refactyl.com) — start free migration.", "url": "https://wpnews.pro/news/our-verifier-was-buggier-than-the-code-it-verified", "canonical_source": "https://dev.to/yoss/our-verifier-was-buggier-than-the-code-it-verified-4231", "published_at": "2026-07-08 18:02:42+00:00", "updated_at": "2026-07-08 18:41:36.568152+00:00", "lang": "en", "topics": ["developer-tools"], "entities": ["Refactyl", "Vue"], "alternates": {"html": "https://wpnews.pro/news/our-verifier-was-buggier-than-the-code-it-verified", "markdown": "https://wpnews.pro/news/our-verifier-was-buggier-than-the-code-it-verified.md", "text": "https://wpnews.pro/news/our-verifier-was-buggier-than-the-code-it-verified.txt", "jsonld": "https://wpnews.pro/news/our-verifier-was-buggier-than-the-code-it-verified.jsonld"}}