I maintain trainproof, a deterministic linter for ML training runs. No model scores your run — every verdict is a rule that either fires or doesn't, and every finding prints the numbers behind it.
Before releasing 0.22.0 I put it through two independent forensic audits, one of them working blind. They took away the single best piece of evidence I had.
Every fault in my validation gallery is injected on purpose — learning rate cranked 100×, fp16 overflow, shuffled labels. Injected faults are clean, and that's the known weakness of any fault-injection study. So the one log I was proudest of was the one nobody touched: a 9.8-hour Coqui XTTS fine-tune from my own work that, I said, "diverged on its own."
It was on the README. It was in two places in my long-form write-up. It was the thing that made the study stand on something real.
The log I shipped stops at step 72,900 of a run that went to 125,039. I had truncated it, published the verdict computed from the truncation, and cited the trainer's own bookkeeping as corroboration — the last BEST MODEL line in the prefix is best_model_49880.pth, which does sit a long way behind the end of the prefix.
The rest of the run was still on disk. In the complete log:
best_model_124700.pth — The bookkeeping I cited as corroboration was corroborating a sentence about the first 58% of a run.
The temptation here is to overcorrect, and that would be the same mistake pointing the other way. So, precisely:
What the evidence supports: the per-micro-batch training display loss ended above its own minimum. That is a property of that one series.
What it does not support: that the model got worse. And it does not support that the model got better either — no audio, MOS or perceptual evaluation of that run was kept. Held-out loss improved. That is all that was measured.
trainproof still returns FAIL on that run, and 0.22.0 does not fix it.
TP-DIVERGE reads one training series. That run's event file carries two:
| series | points | ends |
|---|---|---|
| per-micro-batch loss | 1,251 | 1.88× above its minimum |
| epoch-aggregated loss | 5 | at its minimum |
The reader resolves competing series by density. The 1,251-point series wins; the 5-point series is discarded before any rule sees it. An improving held-out curve sitting in the same file cannot overrule the survivor.
I could have changed the threshold until this particular run went green. That is tuning a rule against a single example until it produces the answer I now prefer, and it would have been the worst available option. So the run still FAILs, the limitation is written into the README, the rules document and the release's machine-readable record, and it is the top item for the next architecture instead of something quietly smoothed over.
Same principle decided each one: remove a claim the tool can't support, or make an advertised behaviour real — never strengthen a claim to close a gap.
A missing optional dependency was a verdict about your work. No sentencepiece installed? trainproof tokenizer exited 1 with a FAIL — the tool reporting its own missing package as a fault in your tokenizer. Now NOT-CHECKED, exit 2. FAIL has to mean something about your run, never "the tool had a problem."
The HuggingFace callback could change the run it was watching. It read the first batches of labels at on_train_begin, which means opening a fresh iterator over your training data. With a map-style and a random sampler drawing from a generator, that consumes generator state and shifts your batch order. A measurement tool perturbing its own experiment. objective_check now defaults to False.
Four evidence strings described something other than what was measured — including a zero-learning-rate finding that printed 100.0% of steps have lr=0 for a run where every value was -1e-4.
No rule ID, threshold or detection predicate changed in 0.22.0. It adds nothing. 497 tests, each correction pinned by a test that fails against the pre-repair code.
No rule in trainproof is calibrated. There is no false-positive rate for any check, because nothing here has been run against a population where the answer is independently known. The gallery is a set of faults I injected on purpose — a regression suite, not a sample.
What the tool offers is determinism and evidence: the same log gives the same verdict, every finding prints the numbers behind it, and a check that could not run says NOT-CHECKED instead of PASS.
Read a finding's evidence as the measurement and its message as an interpretation that may exceed it. That sentence is in the README now because the audits showed it needed to be.
pip install trainproof
trainproof doctor . # discovers and judges every training log it finds
Release 0.22.0 is archived at 10.5281/zenodo.22882964. The full write-up, including what the audits found that I haven't covered here, is on my notebook.
If a deterministic check would have caught a failure that burned your GPU hours, tell me on the repo — it goes in, with credit.
Panagiotis (Panos) Gkilis — founder, BedVibe Studios, the operating brand of BED VIBE GKILIS, a sole proprietorship registered in Norway (org. no. 935 267 897).
Main hub · Portfolio · Work · Research & articles · About · LinkedIn · GitHub · ORCID