{"slug": "a-yaml-comment-truncated-an-ai-coding-success-criterion-and-the-gate-still", "title": "A YAML Comment Truncated an AI Coding Success Criterion — and the Gate Still Passed", "summary": "A developer maintaining the open-source tool spec-lane found that an unquoted YAML plain scalar in a success criterion was silently truncated at a `#` character during parsing, so the text after it never reached the verification gate — and the gate still passed because it compared the already-truncated parsed values. The fix was to quote the criterion as an explicit YAML string so the full text survives parsing; the reproduction is documented in Issue #45.", "body_md": "When I hand implementation work to a coding agent, I want the definition of “done” to exist before the implementation does.\n\nThat usually means writing explicit success criteria, keeping them in a machine-readable form, and checking them again at a verification gate.\n\nI do this in an open-source tool I maintain called `spec-lane`.\n\nWhile dogfooding it, I ran into a small YAML detail with a much larger implication:\n\n**part of a success criterion was interpreted as a YAML comment before it ever reached the verification gate.**\n\nThe more interesting part was that the gate still passed.\n\nThe bug was not in the string comparison itself. The information had already disappeared from the value being compared.\n\nThe minimal case reported in Issue #45 looked like this:\n\n```\nsuccess:\n  - ledger has exactly one PhaseGate row # include the negative case too\n```\n\nThe intent described in the issue included the part after the `#`.\n\nBut this is an unquoted YAML plain scalar.\n\nWith `yaml@2.9.0`, it is parsed roughly as:\n\n```\n{\n  \"success\": [\n    \"ledger has exactly one PhaseGate row\"\n  ]\n}\n```\n\nSo the two representations are different:\n\n```\nsource:\n\nledger has exactly one PhaseGate row # include the negative case too\n\nparsed value:\n\nledger has exactly one PhaseGate row\n```\n\nThe original file has not been modified. The text after `#` is still physically present in the YAML file.\n\nBut it is a comment, so it is not part of the JavaScript value returned by `yaml.parse()`.\n\nThat created a boundary I had not been checking closely enough:\n\n```\nwhat a human sees in the source\n        ↓\n      YAML parse\n        ↓\nwhat downstream code receives\n```\n\nIn `spec-lane`, the success criteria and the verification matrix are separate inputs.\n\nThe success criterion lives in `intent.yaml`.\n\nThe corresponding verification record lives in `verification.yaml`.\n\nFor the reproduction, the matrix contained the already-truncated form:\n\n```\nsuccess_criteria_matrix:\n  - criterion: ledger has exactly one PhaseGate row\n    covered_by: test\n    evidence: test/ledger.test.ts::records PhaseGate\n    negation_test: no PhaseGate row fails\n```\n\nOne important caveat: those `evidence` and `negation_test` values are declarations in the reproduction fixture.\n\nFor this reproduction, I did not create and execute the referenced ledger test and then prove that its behavior matched those strings.\n\nThe purpose here was to reproduce the gate behavior.\n\nI built the CLI from the revision immediately before PR #48's fix and ran the reproduction in a temporary directory.\n\nWith the unquoted success criterion and the shortened matrix entry:\n\n``` bash\n$ lane validate ...\n\nintent.yaml is valid (phase=3_implement).\n```\n\nExit code:\n\n```\n0\n```\n\nThen:\n\n``` php\n$ lane advance ... --phase 4_verify\n\nAdvanced ...: 3_implement -> 4_verify\n```\n\nAgain:\n\n```\nexit code: 0\n```\n\nThe verification phase was entered.\n\nAt that point, the two strings reaching the gate were effectively:\n\n```\nsuccess:\n\nledger has exactly one PhaseGate row\n\nmatrix criterion:\n\nledger has exactly one PhaseGate row\n```\n\nThey matched.\n\nSo the gate passed.\n\nThe important distinction is this:\n\n**the verification gate did not remove the text after `#`.**\n\nThat had already happened at the earlier boundary:\n\n```\nYAML source\n    ↓\nparsed JS value\n```\n\nThe gate was comparing the values it had been given, and those values were equal.\n\nThe relevant code paths are in:\n\n`packages/cli/src/intent-store.ts`` packages/core/src/gate.ts`\nThe full reproduction and investigation are documented in Issue #45:\n\n[https://github.com/shiki-yusuke/spec-lane/issues/45](https://github.com/shiki-yusuke/spec-lane/issues/45)\n\nI then changed only the success criterion so the full text became an explicit YAML string:\n\n```\nsuccess:\n  - \"ledger has exactly one PhaseGate row # include the negative case too\"\n```\n\nNow the parsed value retains everything:\n\n```\nsuccess:\n\nledger has exactly one PhaseGate row # include the negative case too\n\nmatrix criterion:\n\nledger has exactly one PhaseGate row\n```\n\nWith the same pre-fix CLI, both `validate` and `advance` now failed with exit code `3`.\n\nThe gate reported that there was no matrix row corresponding to the full success criterion.\n\nThe phase remained `3_implement`.\n\nSo the control case looked like this:\n\n| Success criterion | Matrix | validate | advance | \n|---|---|---|---|\n| Plain scalar; text after `#` is a comment | Short value | 0 | 0 | \n| Quoted; full value is preserved | Short value | 3 | 3 | \n\nAt least in this fixture, the result did not come from the gate failing to run.\n\nThe value reaching the gate changed.\n\nThat changed the outcome.\n\nThe schema for this part of the success criteria is intentionally simple:\n\n```\nsuccess: z.array(z.string()).min(1)\n```\n\nThe shortened result:\n\n```\nledger has exactly one PhaseGate row\n```\n\nis still a perfectly valid string.\n\nSo after parsing, these two source forms can result in the same value:\n\n```\n- ledger has exactly one PhaseGate row\n```\n\nand:\n\n```\n- ledger has exactly one PhaseGate row # include the negative case too\n```\n\nIf a validator only receives the parsed value, it can no longer tell whether:\n\nThat distinction led me to separate several different claims that are easy to blur together:\n\n```\nthe value has the correct shape\n≠\nthe original source expression was preserved\n```\n\nAnd there are further boundaries after that:\n\n```\nthe string was preserved\n≠\nthe string correctly represents human intent\n```\n\nLikewise:\n\n```\nan evidence field contains a test name\n≠\nthat test was actually executed and proved the claim\n```\n\nA schema validator can validate the structure it receives.\n\nIt cannot automatically recover information that disappeared before that boundary.\n\nPR #48 did not change the success-criteria comparison itself.\n\nInstead, the fix moved the check to the `intent.yaml` reading boundary.\n\nThe reason is straightforward: once only the parsed value remains, there is not enough information to distinguish the two source forms.\n\nThe implementation now roughly does this:\n\n`PLAIN` scalars.`#` on the same line.\nThe implementation is here:\n\nThe check deliberately does not rely only on the AST node's `.comment` property.\n\nOne reason is that certain anchor forms can associate a comment with a different node. The implementation therefore combines AST type and range information with the original source text.\n\nWith the v0.11.0 source, the same reproduction now behaves like this:\n\n| Success criterion | Matrix | validate | advance | \n|---|---|---|---|\n| Plain scalar + inline comment | Short value | 2 | 2 | \n| Quoted full value | Full value | 0 | 0 | \n\nIn the rejected case, the phase remains `3_implement`.\n\nIf a literal `#` belongs in the success criterion, it can be made explicit by quoting the string:\n\n```\nsuccess:\n  - \"ledger has exactly one PhaseGate row # include the negative case too\"\n```\n\nThe fix shipped in `spec-lane` v0.11.0:\n\n[https://github.com/shiki-yusuke/spec-lane/releases/tag/v0.11.0](https://github.com/shiki-yusuke/spec-lane/releases/tag/v0.11.0)\n\nPR #48 contains the implementation and regression tests:\n\n[https://github.com/shiki-yusuke/spec-lane/pull/48](https://github.com/shiki-yusuke/spec-lane/pull/48)\n\nThe fix is intentionally narrower than that.\n\nIt also has a known over-detection case.\n\nConsider:\n\n```\nintent:\n  business_goal: shared text # unrelated comment\n  success:\n    - \"shared text\"\n```\n\nNothing was truncated from the quoted success criterion.\n\nBut the current check can still reject this document because another commented plain scalar has the same parsed value as the success criterion.\n\nThe implementation does not prove that the two occurrences share the same semantic origin.\n\nThat limitation is preserved in a regression test rather than hidden.\n\nSo I would not describe the fix as:\n\nspec-lane now understands whether the author's full intent has been preserved.\n\nIt does not.\n\nThe design is closer to:\n\nif the source contains a value that could have been silently shortened before it reaches a success criterion, stop instead of guessing.\n\nThat is a fail-closed choice.\n\nIt trades some false positives for avoiding this known silent-truncation path.\n\nThis incident changed how I think about verification in AI-assisted development.\n\nIt is tempting to think about the pipeline as:\n\n```\nspecification\n    ↓\nimplementation\n    ↓\nverification\n```\n\nBut in practice there are more boundaries:\n\n```\nHuman expression\nwhat someone is trying to achieve\n        ↓\nSpec source\nwhat was actually written\n        ↓\nParsed representation\nwhat the tool interpreted\n        ↓\nVerification\nwhat the gate compared\n        ↓\nExecution evidence\nwhat actually ran or was observed\n        ↓\nAcceptance\nwhy the result was accepted\n```\n\nThe failure in this post was mostly at:\n\n```\nSpec source\n↓\nParsed representation\n```\n\nEverything after that can behave consistently and still verify less than a human reader thought had been written.\n\nThe same distinction appears elsewhere.\n\nA test name written into a verification matrix is not the same thing as evidence that the test actually ran.\n\nA command exiting successfully is not necessarily the same thing as the intended property being proved.\n\nAnd a schema-valid specification is not necessarily the same thing as preserving every meaningful part of its source representation.\n\nMachine-readable specifications are useful.\n\nBut once a workflow starts treating “the gate passed” as evidence of completion, I also want to know:\n\n**What exactly reached the gate, where did that value come from, and what transformations happened before it got there?**\n\nIn this case, one small YAML line was enough to expose that boundary.\n\n**AI-assisted disclosure:** I used AI tools to help draft, translate, and edit this article. The reproduction, source inspection, and factual claims were checked against the linked issue, pull request, release, and the evidence collected during the investigation.", "url": "https://wpnews.pro/news/a-yaml-comment-truncated-an-ai-coding-success-criterion-and-the-gate-still", "canonical_source": "https://dev.to/shikiyusuke/a-yaml-comment-truncated-an-ai-coding-success-criterion-and-the-gate-still-passed-15jg", "published_at": "2026-09-29 01:37:56+00:00", "updated_at": "2026-09-29 01:48:38.233677+00:00", "lang": "en", "topics": ["ai-agents", "developer-tools", "ai-tools"], "entities": ["spec-lane", "yaml@2.9.0", "Issue #45", "PR #48", "shiki-yusuke"], "also_reported_by": [], "alternates": {"html": "https://wpnews.pro/news/a-yaml-comment-truncated-an-ai-coding-success-criterion-and-the-gate-still", "markdown": "https://wpnews.pro/news/a-yaml-comment-truncated-an-ai-coding-success-criterion-and-the-gate-still.md", "text": "https://wpnews.pro/news/a-yaml-comment-truncated-an-ai-coding-success-criterion-and-the-gate-still.txt", "jsonld": "https://wpnews.pro/news/a-yaml-comment-truncated-an-ai-coding-success-criterion-and-the-gate-still.jsonld"}}