# I audited my own ML linter and had to withdraw its best evidence > Source: > Published: 2026-09-21 20:54:14+00:00 I maintain [trainproof](https://github.com/Mormolykos/trainproof), a deterministic linter for ML training runs. No model scores your run — every verdict is a rule that either fires or doesn't, and every finding prints the numbers behind it. Before releasing 0.22.0 I put it through two independent forensic audits, one of them working blind. **They took away the single best piece of evidence I had.** Every fault in my validation gallery is injected on purpose — learning rate cranked 100×, fp16 overflow, shuffled labels. Injected faults are clean, and that's the known weakness of any fault-injection study. So the one log I was proudest of was the one nobody touched: a 9.8-hour Coqui XTTS fine-tune from my own work that, I said, "diverged on its own." It was on the README. It was in two places in my long-form write-up. It was the thing that made the study stand on something real. The log I shipped stops at **step 72,900 of a run that went to 125,039.** I had truncated it, published the verdict computed from the truncation, and cited the trainer's own bookkeeping as corroboration — the last `BEST MODEL` line in the prefix is `best_model_49880.pth`, which does sit a long way behind the end of the prefix. The rest of the run was still on disk. In the complete log: `best_model_124700.pth` — The bookkeeping I cited as corroboration was corroborating a sentence about the first 58% of a run. The temptation here is to overcorrect, and that would be the same mistake pointing the other way. So, precisely: **What the evidence supports:** the per-micro-batch training display loss ended above its own minimum. That is a property of that one series. **What it does not support:** that the model got worse. And it does *not* support that the model got better either — no audio, MOS or perceptual evaluation of that run was kept. Held-out *loss* improved. That is all that was measured. **trainproof still returns FAIL on that run, and 0.22.0 does not fix it.** `TP-DIVERGE` reads one training series. That run's event file carries two: | series | points | ends | |---|---|---| | per-micro-batch loss | 1,251 | 1.88× above its minimum | | epoch-aggregated loss | 5 | **at** its minimum | The reader resolves competing series by density. The 1,251-point series wins; the 5-point series is discarded before any rule sees it. An improving held-out curve sitting in the same file cannot overrule the survivor. I could have changed the threshold until this particular run went green. That is tuning a rule against a single example until it produces the answer I now prefer, and it would have been the worst available option. So the run still FAILs, the limitation is written into the README, the rules document and the release's machine-readable record, and it is the top item for the next architecture instead of something quietly smoothed over. Same principle decided each one: **remove a claim the tool can't support, or make an advertised behaviour real — never strengthen a claim to close a gap.** **A missing optional dependency was a verdict about your work.** No `sentencepiece` installed? `trainproof tokenizer` exited 1 with a FAIL — the tool reporting *its own* missing package as a fault in your tokenizer. Now `NOT-CHECKED`, exit 2. FAIL has to mean something about your run, never "the tool had a problem." **The HuggingFace callback could change the run it was watching.** It read the first batches of labels at `on_train_begin`, which means opening a fresh iterator over your training dataloader. With a map-style loader and a random sampler drawing from a generator, that consumes generator state and shifts your batch order. A measurement tool perturbing its own experiment. `objective_check` now defaults to `False`. **Four evidence strings described something other than what was measured** — including a zero-learning-rate finding that printed `100.0% of steps have lr=0` for a run where every value was `-1e-4`. No rule ID, threshold or detection predicate changed in 0.22.0. It adds nothing. 497 tests, each correction pinned by a test that fails against the pre-repair code. **No rule in trainproof is calibrated.** There is no false-positive rate for any check, because nothing here has been run against a population where the answer is independently known. The gallery is a set of faults I injected on purpose — a regression suite, not a sample. What the tool offers is determinism and evidence: the same log gives the same verdict, every finding prints the numbers behind it, and a check that could not run says `NOT-CHECKED` instead of `PASS`. Read a finding's *evidence* as the measurement and its *message* as an interpretation that may exceed it. That sentence is in the README now because the audits showed it needed to be. ``` pip install trainproof trainproof doctor . # discovers and judges every training log it finds ``` Release 0.22.0 is archived at [10.5281/zenodo.22882964](https://doi.org/10.5281/zenodo.22882964). The full write-up, including what the audits found that I haven't covered here, is [on my notebook](https://ai.bedvibe.studio/trainproof/#withdrawn). If a deterministic check would have caught a failure that burned your GPU hours, tell me on the repo — it goes in, with credit. **Panagiotis (Panos) Gkilis** — founder, BedVibe Studios, the operating brand of BED VIBE GKILIS, a sole proprietorship registered in Norway (org. no. 935 267 897). [Main hub](https://bedvibe.studio/) · [Portfolio](https://tts.bedvibe.studio/portfolio/) · [Work](https://ai.bedvibe.studio/work/) · [Research & articles](https://ai.bedvibe.studio/) · [About](https://bedvibe.studio/panagiotis-panos-gkilis/) · [LinkedIn](https://www.linkedin.com/in/panagiotis-gkilis-57995117b/) · [GitHub](https://github.com/Mormolykos) · [ORCID](https://orcid.org/0009-0007-3805-170X)