I audited my own ML linter and had to withdraw its best evidence

A developer maintaining trainproof, a deterministic linter for ML training runs, withdrew the strongest evidence behind the tool's validation gallery after two independent forensic audits found the cited log had been truncated at step 72,900 of a 125,039-step Coqui XTTS fine-tune. The complete log showed the run's best checkpoint at step 124,700, contradicting the bookkeeping cited as corroboration, and the tool still returns FAIL on that run because its TP-DIVERGE rule reads only the denser per-micro-batch loss series and discards the improving epoch-aggregated curve. The 0.22.0 release documents the limitation, reclassifies a missing sentencepiece dependency as NOT-CHECKED rather than FAIL, and disables the HuggingFace callback's objective_check by default to stop the tool from perturbing the runs it measures.

I maintain trainproof https://github.com/Mormolykos/trainproof , a deterministic linter for ML training runs. No model scores your run — every verdict is a rule that either fires or doesn't, and every finding prints the numbers behind it. Before releasing 0.22.0 I put it through two independent forensic audits, one of them working blind. They took away the single best piece of evidence I had. Every fault in my validation gallery is injected on purpose — learning rate cranked 100×, fp16 overflow, shuffled labels. Injected faults are clean, and that's the known weakness of any fault-injection study. So the one log I was proudest of was the one nobody touched: a 9.8-hour Coqui XTTS fine-tune from my own work that, I said, "diverged on its own." It was on the README. It was in two places in my long-form write-up. It was the thing that made the study stand on something real. The log I shipped stops at step 72,900 of a run that went to 125,039. I had truncated it, published the verdict computed from the truncation, and cited the trainer's own bookkeeping as corroboration — the last BEST MODEL line in the prefix is best model 49880.pth , which does sit a long way behind the end of the prefix. The rest of the run was still on disk. In the complete log: best model 124700.pth — The bookkeeping I cited as corroboration was corroborating a sentence about the first 58% of a run. The temptation here is to overcorrect, and that would be the same mistake pointing the other way. So, precisely: What the evidence supports: the per-micro-batch training display loss ended above its own minimum. That is a property of that one series. What it does not support: that the model got worse. And it does not support that the model got better either — no audio, MOS or perceptual evaluation of that run was kept. Held-out loss improved. That is all that was measured. trainproof still returns FAIL on that run, and 0.22.0 does not fix it. TP-DIVERGE reads one training series. That run's event file carries two: | series | points | ends | |---|---|---| | per-micro-batch loss | 1,251 | 1.88× above its minimum | | epoch-aggregated loss | 5 | at its minimum | The reader resolves competing series by density. The 1,251-point series wins; the 5-point series is discarded before any rule sees it. An improving held-out curve sitting in the same file cannot overrule the survivor. I could have changed the threshold until this particular run went green. That is tuning a rule against a single example until it produces the answer I now prefer, and it would have been the worst available option. So the run still FAILs, the limitation is written into the README, the rules document and the release's machine-readable record, and it is the top item for the next architecture instead of something quietly smoothed over. Same principle decided each one: remove a claim the tool can't support, or make an advertised behaviour real — never strengthen a claim to close a gap. A missing optional dependency was a verdict about your work. No sentencepiece installed? trainproof tokenizer exited 1 with a FAIL — the tool reporting its own missing package as a fault in your tokenizer. Now NOT-CHECKED , exit 2. FAIL has to mean something about your run, never "the tool had a problem." The HuggingFace callback could change the run it was watching. It read the first batches of labels at on train begin , which means opening a fresh iterator over your training dataloader. With a map-style loader and a random sampler drawing from a generator, that consumes generator state and shifts your batch order. A measurement tool perturbing its own experiment. objective check now defaults to False . Four evidence strings described something other than what was measured — including a zero-learning-rate finding that printed 100.0% of steps have lr=0 for a run where every value was -1e-4 . No rule ID, threshold or detection predicate changed in 0.22.0. It adds nothing. 497 tests, each correction pinned by a test that fails against the pre-repair code. No rule in trainproof is calibrated. There is no false-positive rate for any check, because nothing here has been run against a population where the answer is independently known. The gallery is a set of faults I injected on purpose — a regression suite, not a sample. What the tool offers is determinism and evidence: the same log gives the same verdict, every finding prints the numbers behind it, and a check that could not run says NOT-CHECKED instead of PASS . Read a finding's evidence as the measurement and its message as an interpretation that may exceed it. That sentence is in the README now because the audits showed it needed to be. pip install trainproof trainproof doctor . discovers and judges every training log it finds Release 0.22.0 is archived at 10.5281/zenodo.22882964 https://doi.org/10.5281/zenodo.22882964 . The full write-up, including what the audits found that I haven't covered here, is on my notebook https://ai.bedvibe.studio/trainproof/ withdrawn . If a deterministic check would have caught a failure that burned your GPU hours, tell me on the repo — it goes in, with credit. Panagiotis Panos Gkilis — founder, BedVibe Studios, the operating brand of BED VIBE GKILIS, a sole proprietorship registered in Norway org. no. 935 267 897 . Main hub https://bedvibe.studio/ · Portfolio https://tts.bedvibe.studio/portfolio/ · Work https://ai.bedvibe.studio/work/ · Research & articles https://ai.bedvibe.studio/ · About https://bedvibe.studio/panagiotis-panos-gkilis/ · LinkedIn https://www.linkedin.com/in/panagiotis-gkilis-57995117b/ · GitHub https://github.com/Mormolykos · ORCID https://orcid.org/0009-0007-3805-170X