# A YAML Comment Truncated an AI Coding Success Criterion — and the Gate Still Passed > Source: > Published: 2026-09-29 01:37:56+00:00 When I hand implementation work to a coding agent, I want the definition of “done” to exist before the implementation does. That usually means writing explicit success criteria, keeping them in a machine-readable form, and checking them again at a verification gate. I do this in an open-source tool I maintain called `spec-lane`. While dogfooding it, I ran into a small YAML detail with a much larger implication: **part of a success criterion was interpreted as a YAML comment before it ever reached the verification gate.** The more interesting part was that the gate still passed. The bug was not in the string comparison itself. The information had already disappeared from the value being compared. The minimal case reported in Issue #45 looked like this: ``` success: - ledger has exactly one PhaseGate row # include the negative case too ``` The intent described in the issue included the part after the `#`. But this is an unquoted YAML plain scalar. With `yaml@2.9.0`, it is parsed roughly as: ``` { "success": [ "ledger has exactly one PhaseGate row" ] } ``` So the two representations are different: ``` source: ledger has exactly one PhaseGate row # include the negative case too parsed value: ledger has exactly one PhaseGate row ``` The original file has not been modified. The text after `#` is still physically present in the YAML file. But it is a comment, so it is not part of the JavaScript value returned by `yaml.parse()`. That created a boundary I had not been checking closely enough: ``` what a human sees in the source ↓ YAML parse ↓ what downstream code receives ``` In `spec-lane`, the success criteria and the verification matrix are separate inputs. The success criterion lives in `intent.yaml`. The corresponding verification record lives in `verification.yaml`. For the reproduction, the matrix contained the already-truncated form: ``` success_criteria_matrix: - criterion: ledger has exactly one PhaseGate row covered_by: test evidence: test/ledger.test.ts::records PhaseGate negation_test: no PhaseGate row fails ``` One important caveat: those `evidence` and `negation_test` values are declarations in the reproduction fixture. For this reproduction, I did not create and execute the referenced ledger test and then prove that its behavior matched those strings. The purpose here was to reproduce the gate behavior. I built the CLI from the revision immediately before PR #48's fix and ran the reproduction in a temporary directory. With the unquoted success criterion and the shortened matrix entry: ``` bash $ lane validate ... intent.yaml is valid (phase=3_implement). ``` Exit code: ``` 0 ``` Then: ``` php $ lane advance ... --phase 4_verify Advanced ...: 3_implement -> 4_verify ``` Again: ``` exit code: 0 ``` The verification phase was entered. At that point, the two strings reaching the gate were effectively: ``` success: ledger has exactly one PhaseGate row matrix criterion: ledger has exactly one PhaseGate row ``` They matched. So the gate passed. The important distinction is this: **the verification gate did not remove the text after `#`.** That had already happened at the earlier boundary: ``` YAML source ↓ parsed JS value ``` The gate was comparing the values it had been given, and those values were equal. The relevant code paths are in: `packages/cli/src/intent-store.ts`` packages/core/src/gate.ts` The full reproduction and investigation are documented in Issue #45: [https://github.com/shiki-yusuke/spec-lane/issues/45](https://github.com/shiki-yusuke/spec-lane/issues/45) I then changed only the success criterion so the full text became an explicit YAML string: ``` success: - "ledger has exactly one PhaseGate row # include the negative case too" ``` Now the parsed value retains everything: ``` success: ledger has exactly one PhaseGate row # include the negative case too matrix criterion: ledger has exactly one PhaseGate row ``` With the same pre-fix CLI, both `validate` and `advance` now failed with exit code `3`. The gate reported that there was no matrix row corresponding to the full success criterion. The phase remained `3_implement`. So the control case looked like this: | Success criterion | Matrix | validate | advance | |---|---|---|---| | Plain scalar; text after `#` is a comment | Short value | 0 | 0 | | Quoted; full value is preserved | Short value | 3 | 3 | At least in this fixture, the result did not come from the gate failing to run. The value reaching the gate changed. That changed the outcome. The schema for this part of the success criteria is intentionally simple: ``` success: z.array(z.string()).min(1) ``` The shortened result: ``` ledger has exactly one PhaseGate row ``` is still a perfectly valid string. So after parsing, these two source forms can result in the same value: ``` - ledger has exactly one PhaseGate row ``` and: ``` - ledger has exactly one PhaseGate row # include the negative case too ``` If a validator only receives the parsed value, it can no longer tell whether: That distinction led me to separate several different claims that are easy to blur together: ``` the value has the correct shape ≠ the original source expression was preserved ``` And there are further boundaries after that: ``` the string was preserved ≠ the string correctly represents human intent ``` Likewise: ``` an evidence field contains a test name ≠ that test was actually executed and proved the claim ``` A schema validator can validate the structure it receives. It cannot automatically recover information that disappeared before that boundary. PR #48 did not change the success-criteria comparison itself. Instead, the fix moved the check to the `intent.yaml` reading boundary. The reason is straightforward: once only the parsed value remains, there is not enough information to distinguish the two source forms. The implementation now roughly does this: `PLAIN` scalars.`#` on the same line. The implementation is here: The check deliberately does not rely only on the AST node's `.comment` property. One reason is that certain anchor forms can associate a comment with a different node. The implementation therefore combines AST type and range information with the original source text. With the v0.11.0 source, the same reproduction now behaves like this: | Success criterion | Matrix | validate | advance | |---|---|---|---| | Plain scalar + inline comment | Short value | 2 | 2 | | Quoted full value | Full value | 0 | 0 | In the rejected case, the phase remains `3_implement`. If a literal `#` belongs in the success criterion, it can be made explicit by quoting the string: ``` success: - "ledger has exactly one PhaseGate row # include the negative case too" ``` The fix shipped in `spec-lane` v0.11.0: [https://github.com/shiki-yusuke/spec-lane/releases/tag/v0.11.0](https://github.com/shiki-yusuke/spec-lane/releases/tag/v0.11.0) PR #48 contains the implementation and regression tests: [https://github.com/shiki-yusuke/spec-lane/pull/48](https://github.com/shiki-yusuke/spec-lane/pull/48) The fix is intentionally narrower than that. It also has a known over-detection case. Consider: ``` intent: business_goal: shared text # unrelated comment success: - "shared text" ``` Nothing was truncated from the quoted success criterion. But the current check can still reject this document because another commented plain scalar has the same parsed value as the success criterion. The implementation does not prove that the two occurrences share the same semantic origin. That limitation is preserved in a regression test rather than hidden. So I would not describe the fix as: spec-lane now understands whether the author's full intent has been preserved. It does not. The design is closer to: if the source contains a value that could have been silently shortened before it reaches a success criterion, stop instead of guessing. That is a fail-closed choice. It trades some false positives for avoiding this known silent-truncation path. This incident changed how I think about verification in AI-assisted development. It is tempting to think about the pipeline as: ``` specification ↓ implementation ↓ verification ``` But in practice there are more boundaries: ``` Human expression what someone is trying to achieve ↓ Spec source what was actually written ↓ Parsed representation what the tool interpreted ↓ Verification what the gate compared ↓ Execution evidence what actually ran or was observed ↓ Acceptance why the result was accepted ``` The failure in this post was mostly at: ``` Spec source ↓ Parsed representation ``` Everything after that can behave consistently and still verify less than a human reader thought had been written. The same distinction appears elsewhere. A test name written into a verification matrix is not the same thing as evidence that the test actually ran. A command exiting successfully is not necessarily the same thing as the intended property being proved. And a schema-valid specification is not necessarily the same thing as preserving every meaningful part of its source representation. Machine-readable specifications are useful. But once a workflow starts treating “the gate passed” as evidence of completion, I also want to know: **What exactly reached the gate, where did that value come from, and what transformations happened before it got there?** In this case, one small YAML line was enough to expose that boundary. **AI-assisted disclosure:** I used AI tools to help draft, translate, and edit this article. The reproduction, source inspection, and factual claims were checked against the linked issue, pull request, release, and the evidence collected during the investigation.