{"slug": "the-same-bug-four-times-three-of-them-mine", "title": "The Same Bug, Four Times, Three of Them Mine", "summary": "An engineer at Bedvibe Studio, writing under the alias Mormolykos, has released a new open-source library called trainproof after repeatedly shipping the same validation bug in different forms. The tool, which evaluates machine learning training runs and other artifacts, previously reported unevaluated checks as passed, leading to false positives and missed failures. The new library introduces a state model that distinguishes between checked, not-checked, and out-of-scope outcomes, with explicit owners for each gap, to prevent silent misreporting.", "body_md": "*Originally published at ai.bedvibe.studio.*\n\nMost validation tooling has two states: it passed, or it failed. Everything that was not actually evaluated has to be forced into one of them — and it is wrong in both directions.\n\nI did not work that out from first principles. I worked it out by shipping the same defect four times.\n\nIn [trainproof](https://github.com/Mormolykos/trainproof), a training run whose loss was exactly `0.0`\n\non every step returned **PASS**.\n\nEvery loss-shape check in that tool is guarded against dividing by zero. A curve that is identically zero trips every guard, so all of them skipped — silently. And then the report listed those same skipped checks as having *run*. A run that learned nothing passed, accompanied by a list of the checks that had cleared it.\n\nThat is the whole problem in one artifact. The tool had no way to say *I could not judge this*, so the absence of a judgement rendered as a favourable one.\n\nThe fix for that was a report field listing which checks ran and which did not. Good. Then a nastier version turned up somewhere else.\n\n`trainproof doctor`\n\nwalks a directory twice — once to discover candidate logs, once to judge them. The judging pass reported anything it could not read. The discovery pass had `except Exception: pass`\n\n.\n\nSo a file that raised *while being found* never became a candidate, and never appeared in the report at all. Plainly visible on disk. Absent from the output. Indistinguishable from a file that passed.\n\nSame failure as the first, one loop earlier than I had been looking. That is when I stopped treating it as a bug.\n\nI wrote the first two up publicly. Someone working in infrastructure compliance replied with the identical shape from a domain I know nothing about.\n\nA compliance framework document is mostly prose. Most of it describes things no generated artifact can satisfy or violate — staff training, review procedures, who signs what. Only a fraction maps to something a machine can check.\n\nThe trap is reporting against the framework's name. Do that and everything unevaluated looks identical to everything that passed, and the ninety per cent that was never in scope disappears from the output entirely.\n\nThat exchange is why this became a library rather than another `trainproof`\n\nfeature. One person hitting a shape twice is a habit. Two people hitting it in unrelated domains is a primitive.\n\nThen I ran an evaluation harness of my own over a retrieval experiment and found three instances in a single afternoon.\n\nIt recorded model refusals under a failure type asserting an ordering the model had never given. It scored ten refusals as **correct**, because the expected phrase appeared inside the sentence explaining what could not be determined. And it missed eight correct answers because its negative pattern required a comma.\n\nOne absent value. Wrong in both directions. Inside the instrument I was using to judge my own hypothesis.\n\nThree instead of two — checked, could-not-check, never-in-scope — and eight once you ask two more questions of each gap: **who can fix it**, and **can it ever change**.\n\n| State | Owner | What it means |\n|---|---|---|\n`CHECKED` |\n— | a determination was made; the verdict is separate |\n`NOT_CHECKED / DATA_DEGENERATE` |\nthe data | the signal is present and unusable |\n`NOT_CHECKED / CHECKER_FAILED` |\nyour tooling | the checker raised, timed out, or could not observe |\n`NOT_CHECKED / WAIVED` |\na named person | in scope, deliberately not evaluated, accepted |\n`NOT_CHECKED / PREREQUISITE_FAILED` |\nanother target | something upstream failed first |\n`OUT_OF_SCOPE / CALLER` |\nthe caller | not requested |\n`OUT_OF_SCOPE / DATA_TRANSIENT` |\nthe deployment | does not apply yet\n|\n`OUT_OF_SCOPE / DATA_PERMANENT` |\nnobody | no artifact of this kind can ever evidence it |\n\n**Coverage is not verdict.** `CHECKED`\n\nis not a result — it says a determination was made, not what it was. The verdict vocabulary belongs to your domain (pass/warn/fail, compliant/non-compliant) and hangs off `CHECKED`\n\nrather than sitting beside the not-checked states. Collapsing those two axes is failure one above. The constructor enforces the separation: a gap cannot carry a verdict, and a checked record cannot carry a skip reason.\n\nBoris Teplitsky, from the compliance side, put three corrections into the schema that I would not have found:\n\n**Permanence is relative to a target, never absolute.** \"Nobody, never\" is not a property of a control — it is a property of pairing that control with a kind of artifact. A Kubernetes control is permanently out of scope only while the target has no Kubernetes; change the target and it becomes a row. So `OUT_OF_SCOPE / DATA_PERMANENT`\n\nnow *requires* a `permanent_wrt`\n\nfield naming the reference. Unqualified, two reports on the same framework disagree and both are correct.\n\n**Rows for the checkable subset, one count for the rest.** A framework document is hundreds of pages of which a few paragraphs concern anything an artifact can evidence. One row each makes the report noise. So the excluded corpus can be a single counted rule — and the count cannot be stated without the rule that produced it, and it enters the denominator, so excluding 412 of 415 reports 99% rather than a flattering silence.\n\n**The taxonomy starts after the unit exists.** In a linter, a row is a check somebody wrote. In compliance, turning prose into checkable requirements is the hard part and it is a judgment call. Nothing in the library governs that, and it should not read as if the rows arrive by themselves.\n\nThat last one is scope, not code, and it was the correction I was most wrong about.\n\nRunning the README example through the new bulk path printed `413 out of scope (100% of all targets)`\n\nwhile one target had in fact been checked — `.0%`\n\nrounded 99.5 up. A reader takes \"100% out of scope\" as \"nothing was measured.\"\n\nThat is this library's own failure mode, committed by its own renderer, one layer above the schema it protects. Fixed both directions: 100% only when everything is excluded, 0% only when nothing is.\n\nMIT, no dependencies, 74 tests: [github.com/Mormolykos/notchecked](https://github.com/Mormolykos/notchecked)\n\nIf you have hit this shape in a fifth domain, I would like to hear it — the case list is a test file now, so a new case goes in with credit.", "url": "https://wpnews.pro/news/the-same-bug-four-times-three-of-them-mine", "canonical_source": "https://dev.to/bedvibe_studios/the-same-bug-four-times-three-of-them-mine-4b78", "published_at": "2026-08-25 02:20:15+00:00", "updated_at": "2026-08-25 03:13:13.866613+00:00", "lang": "en", "topics": ["mlops", "developer-tools", "machine-learning"], "entities": ["Mormolykos", "trainproof", "Bedvibe Studio"], "alternates": {"html": "https://wpnews.pro/news/the-same-bug-four-times-three-of-them-mine", "markdown": "https://wpnews.pro/news/the-same-bug-four-times-three-of-them-mine.md", "text": "https://wpnews.pro/news/the-same-bug-four-times-three-of-them-mine.txt", "jsonld": "https://wpnews.pro/news/the-same-bug-four-times-three-of-them-mine.jsonld"}}