{"slug": "i-audited-my-own-ml-linter-and-had-to-withdraw-its-best-evidence", "title": "I audited my own ML linter and had to withdraw its best evidence", "summary": "A developer maintaining trainproof, a deterministic linter for ML training runs, withdrew the strongest evidence behind the tool's validation gallery after two independent forensic audits found the cited log had been truncated at step 72,900 of a 125,039-step Coqui XTTS fine-tune. The complete log showed the run's best checkpoint at step 124,700, contradicting the bookkeeping cited as corroboration, and the tool still returns FAIL on that run because its TP-DIVERGE rule reads only the denser per-micro-batch loss series and discards the improving epoch-aggregated curve. The 0.22.0 release documents the limitation, reclassifies a missing sentencepiece dependency as NOT-CHECKED rather than FAIL, and disables the HuggingFace callback's objective_check by default to stop the tool from perturbing the runs it measures.", "body_md": "I maintain [trainproof](https://github.com/Mormolykos/trainproof), a deterministic linter for ML training runs. No model scores your run — every verdict is a rule that either fires or doesn't, and every finding prints the numbers behind it.\n\nBefore releasing 0.22.0 I put it through two independent forensic audits, one of them working blind. **They took away the single best piece of evidence I had.**\n\nEvery fault in my validation gallery is injected on purpose — learning rate cranked 100×, fp16 overflow, shuffled labels. Injected faults are clean, and that's the known weakness of any fault-injection study. So the one log I was proudest of was the one nobody touched: a 9.8-hour Coqui XTTS fine-tune from my own work that, I said, \"diverged on its own.\"\n\nIt was on the README. It was in two places in my long-form write-up. It was the thing that made the study stand on something real.\n\nThe log I shipped stops at **step 72,900 of a run that went to 125,039.** I had truncated it, published the verdict computed from the truncation, and cited the trainer's own bookkeeping as corroboration — the last `BEST MODEL` line in the prefix is `best_model_49880.pth`, which does sit a long way behind the end of the prefix.\n\nThe rest of the run was still on disk. In the complete log:\n\n`best_model_124700.pth` — The bookkeeping I cited as corroboration was corroborating a sentence about the first 58% of a run.\n\nThe temptation here is to overcorrect, and that would be the same mistake pointing the other way. So, precisely:\n\n**What the evidence supports:** the per-micro-batch training display loss ended above its own minimum. That is a property of that one series.\n\n**What it does not support:** that the model got worse. And it does *not* support that the model got better either — no audio, MOS or perceptual evaluation of that run was kept. Held-out *loss* improved. That is all that was measured.\n\n**trainproof still returns FAIL on that run, and 0.22.0 does not fix it.**\n\n`TP-DIVERGE` reads one training series. That run's event file carries two:\n\n| series | points | ends | \n|---|---|---|\n| per-micro-batch loss | 1,251 | 1.88× above its minimum | \n| epoch-aggregated loss | 5 | **at** its minimum | \n\nThe reader resolves competing series by density. The 1,251-point series wins; the 5-point series is discarded before any rule sees it. An improving held-out curve sitting in the same file cannot overrule the survivor.\n\nI could have changed the threshold until this particular run went green. That is tuning a rule against a single example until it produces the answer I now prefer, and it would have been the worst available option. So the run still FAILs, the limitation is written into the README, the rules document and the release's machine-readable record, and it is the top item for the next architecture instead of something quietly smoothed over.\n\nSame principle decided each one: **remove a claim the tool can't support, or make an advertised behaviour real — never strengthen a claim to close a gap.**\n\n**A missing optional dependency was a verdict about your work.** No `sentencepiece` installed? `trainproof tokenizer` exited 1 with a FAIL — the tool reporting *its own* missing package as a fault in your tokenizer. Now `NOT-CHECKED`, exit 2. FAIL has to mean something about your run, never \"the tool had a problem.\"\n\n**The HuggingFace callback could change the run it was watching.** It read the first batches of labels at `on_train_begin`, which means opening a fresh iterator over your training dataloader. With a map-style loader and a random sampler drawing from a generator, that consumes generator state and shifts your batch order. A measurement tool perturbing its own experiment. `objective_check` now defaults to `False`.\n\n**Four evidence strings described something other than what was measured** — including a zero-learning-rate finding that printed `100.0% of steps have lr=0` for a run where every value was `-1e-4`.\n\nNo rule ID, threshold or detection predicate changed in 0.22.0. It adds nothing. 497 tests, each correction pinned by a test that fails against the pre-repair code.\n\n**No rule in trainproof is calibrated.** There is no false-positive rate for any check, because nothing here has been run against a population where the answer is independently known. The gallery is a set of faults I injected on purpose — a regression suite, not a sample.\n\nWhat the tool offers is determinism and evidence: the same log gives the same verdict, every finding prints the numbers behind it, and a check that could not run says `NOT-CHECKED` instead of `PASS`.\n\nRead a finding's *evidence* as the measurement and its *message* as an interpretation that may exceed it. That sentence is in the README now because the audits showed it needed to be.\n\n```\npip install trainproof\ntrainproof doctor .   # discovers and judges every training log it finds\n```\n\nRelease 0.22.0 is archived at [10.5281/zenodo.22882964](https://doi.org/10.5281/zenodo.22882964). The full write-up, including what the audits found that I haven't covered here, is [on my notebook](https://ai.bedvibe.studio/trainproof/#withdrawn).\n\nIf a deterministic check would have caught a failure that burned your GPU hours, tell me on the repo — it goes in, with credit.\n\n**Panagiotis (Panos) Gkilis** — founder, BedVibe Studios, the operating brand of BED VIBE GKILIS, a sole proprietorship registered in Norway (org. no. 935 267 897).\n\n[Main hub](https://bedvibe.studio/) · [Portfolio](https://tts.bedvibe.studio/portfolio/) · [Work](https://ai.bedvibe.studio/work/) · [Research & articles](https://ai.bedvibe.studio/) · [About](https://bedvibe.studio/panagiotis-panos-gkilis/) · [LinkedIn](https://www.linkedin.com/in/panagiotis-gkilis-57995117b/) · [GitHub](https://github.com/Mormolykos) · [ORCID](https://orcid.org/0009-0007-3805-170X)", "url": "https://wpnews.pro/news/i-audited-my-own-ml-linter-and-had-to-withdraw-its-best-evidence", "canonical_source": "https://dev.to/bedvibe_studios/i-audited-my-own-ml-linter-and-had-to-withdraw-its-best-evidence-1g96", "published_at": "2026-09-21 20:54:14+00:00", "updated_at": "2026-09-21 21:25:11.711386+00:00", "lang": "en", "topics": ["machine-learning", "mlops", "developer-tools", "ai-tools"], "entities": ["trainproof", "Coqui XTTS", "HuggingFace", "sentencepiece", "TP-DIVERGE"], "alternates": {"html": "https://wpnews.pro/news/i-audited-my-own-ml-linter-and-had-to-withdraw-its-best-evidence", "markdown": "https://wpnews.pro/news/i-audited-my-own-ml-linter-and-had-to-withdraw-its-best-evidence.md", "text": "https://wpnews.pro/news/i-audited-my-own-ml-linter-and-had-to-withdraw-its-best-evidence.txt", "jsonld": "https://wpnews.pro/news/i-audited-my-own-ml-linter-and-had-to-withdraw-its-best-evidence.jsonld"}}