Loss Curves Lie: Building a Deterministic Linter for ML Training Runs

Trainproof v0.18.0, a deterministic linter for machine learning training runs, catches training failures such as a cross-entropy ignore_index collision that silently deletes a target from the loss while leaving the loss curve indistinguishable from a correct run (0.0035 vs 0.0034), and a live guardian aborted a diverging Qwen2.5-3B QLoRA at step 20 of 300 in a real demo. The tool, released under MIT by developer Mormolykos, reads five log formats including TensorBoard event files without requiring torch, tensorflow, or tensorboard, and is grounded in an 18-run fault-injection study (Qwen2.5-3B QLoRA, RTX 5080) plus a 9.8-hour Coqui XTTS fine-tune that diverged on its own.

Loss Curves Lie: Building a Deterministic Linter for ML Training Runs A run learning pure noise reduced its loss by 62% and looked textbook-healthy. trainproof catches the training failures that can be caught — and is honest about the ones that can't. trainproof v0.18.0activeDeterministic PASS/WARN/FAIL verdicts across a training run's whole life — before it starts dataset, tokenizer, and an environment preflight that catches a stack which will not import, an unreadable checkpoint, or too little system RAM , a live guardian during training, a post-run autopsy after, and a baseline compare. Reads five log formats including TensorBoard event files, decoded from the wire format with no torch, tensorflow or tensorboard installed. No ML judging ML: every verdict is a rule that fires or doesn't, every finding cites its exact numbers, and a check that could not run is reported as NOT-CHECKED rather than as a pass. - Objective checks: catches a cross-entropy ignore index that is also a valid output class - a collision that silently deletes a target from the loss while leaving the loss curve indistinguishable from a correct run 0.0035 vs 0.0034 - trainproof doctor . — zero-config autopsy: discovers every training log in a directory, judges them, prints a triage-sorted summary - Live guardian: a one-line Hugging Face callback; opt-in stop on fail aborted a diverging Qwen2.5-3B QLoRA at step 20 of 300 in a real demo - Preflight: lints dataset + tokenizer before a single GPU-second malformed JSONL, empty rows, duplicates, missing eos/pad, over-length - compare catches shuffled-labels corruption that single-run loss rules cannot — the relative loss floor sits ~6× a known-good baseline - Stable rule IDs TP-DIVERGE, TP-OVERFIT, TP-DEAD-RUN, … documented in RULES.md, plus --json output built for CI and AI coding agents - CONTRACTS.md pins the promises — exit codes, JSON schema policy, rule-ID stability and a verdict-stability guarantee; SARIF 2.1.0 output turns findings into GitHub PR annotations - A PASS names every check that ran and every check it skipped, with the reason — a skipped check is never reported as a passed one - Grounded in an 18-run fault-injection study Qwen2.5-3B QLoRA, RTX 5080 — six configurations at three seeds each, every real log shipped in-repo — plus a 9.8-hour Coqui XTTS fine-tune that diverged on its own, the first shipped failure nobody injected - Reads five log formats including TensorBoard event files, decoded from the wire format with no tensorflow, tensorboard, protobuf, torch or numpy — validated byte-exact against tensorboard's own EventAccumulator, and the rules are exercised on real runs from three frameworks: HuggingFace, Coqui XTTS and PyTorch Lightning Recent Releases v0.18.0 2026-08-09 — v0.18.0 - the objective release v0.17.0 2026-08-02 — v0.17.0 — the lint gate, and one log that used to vanish v0.16.0 2026-08-02 — v0.16.0 — the rule registry no behaviour change v0.15.0 2026-08-01 — v0.15.0 — the before-the-GPU release I kept losing GPU hours to runs that were already dead. A fine-tune that had quietly gone to NaN somewhere after step 300 and trained on garbage the rest of the night. A run that was training at learning-rate- zero the whole time and updating nothing. A dataset with a few hundred broken rows I only discovered at the end. Every time, the pattern was the same: the run looked like it was working, and nothing told me otherwise until the hours were already spent. So I built a linter for training runs — trainproof — and then I spent days trying to prove it wrong. pip install trainproof trainproof doctor . zero-config: discovers and judges every training log it finds Repo: github.com/Mormolykos/trainproof MIT The philosophy: no ML judging ML The most important decision in trainproof is what it isn't . There's no model scoring your run. No "87%-confidence this looks unhealthy." Every verdict is a deterministic rule that either fires or it doesn't, and every finding cites the exact numbers behind it. When it can't be sure, it says so instead of guessing. That constraint is the whole point. A reliability tool that hallucinates is worse than no tool, because now you don't trust your own alarms. trainproof's engine is IO-free and doesn't even import torch — it reads plain logs and applies rules. You can read every rule and know exactly why it fired. Validating the rules: controlled fault injection You validate a detector by feeding it faults whose answer you already know, then measuring what it catches. One base setup — a Qwen2.5-3B QLoRA on an RTX 5080 — run six ways: healthy — the control. lr hot — learning rate cranked 100× too high. lr zero — learning rate at zero. fp16 nan — an fp16 overflow that NaNs the loss. bad labels — the dataset's labels shuffled into pure noise. overfit — 64 training samples and many epochs: pure memorisation, with a held-out eval set to prove it. Three seeds each: 18 runs , real logs, all shipped in the repo so you can reproduce every verdict. A nineteenth log ships beside them with no injected fault at all — a 9.8-hour Coqui XTTS fine-tune that diverged on its own. Four of the five failures got caught fast. lr hot spiked the gradient norm to 2,650× the median — flagged in seconds. The NaN and the flatline were trivial. But one config beat the tool completely, and it's the one that taught me the most. The one that fooled it: shuffled labels bad labels was pure garbage — a dataset that cannot be learned , because the labels no longer correspond to the inputs. And that run reduced its loss by 62%. On its own loss curve it looked like textbook-healthy training: a clean downward slope. It was learning absolutely nothing useful — just memorizing the statistics of noise, which any sufficiently large network will happily do. From a single run's loss curve, it is indistinguishable from a real run. No loss-only rule, looking at that run in isolation, can catch it. That's not a gap I papered over — I wrote it straight into the README. It's also why trainproof has a compare mode: the failure becomes obvious the moment you put the run next to a known-good baseline and look at the relative floor it reaches. The corrupted run's loss floor sits in a different regime. One run in isolation lies; two runs side by side tell the truth. What trainproof actually does It covers a run's whole life, and every stage exits with a status code so it drops straight into CI: Before a single GPU-second — Lints the dataset and tokenizer: malformed JSONL with the line number , empty rows, duplicates, missing preflight . eos token , pad == eos , over-length samples. Catch the broken dataset before you rent the GPU. During training — the guardian. A one-line Hugging Face callback. It warns by default; flip on stop on fail and it will abort a doomed run itself. In one guarded demo, it killed a diverging run at step 20 of 300 scheduled steps — so 280/300 ≈93% of the scheduled steps never executed. That's a statement about that run , not a promise about your GPU bill — but a diverging run you stop at step 20 is a diverging run you didn't pay to finish. After — Read the finished log and classify it: diverged, flatlined, NaN'd, spiked, overfitting. epoch and doctor . Versus a baseline — The relative-floor rules that catch the shuffled-labels case. compare . It reads the logs you already produce — Hugging Face trainer state.json , Coqui, plain JSONL/CSV — so there's nothing to instrument. Try it pip install trainproof - Repo with all 18 fault-injection logs, plus the one that broke by itself : https://github.com/Mormolykos/trainproof https://github.com/Mormolykos/trainproof - MIT licensed. If you fine-tune, you've had at least one of these failures. I'd genuinely like to know which failure mode has burned your GPU hours — if a deterministic check would have caught it, tell me on the repo and it goes in, with credit. Update: what came after Since this article was written, trainproof has grown from the experiment above into a fuller tool. The release timeline at the top of this page is generated from GitHub Releases and PyPI, so it is always current — this section covers what changed in kind, not which version you are on: — point it at a directory; it discovers every training log, judges them all, and prints a triage-sorted summary. The six configurations above are now one command. trainproof doctor . Guardian telemetry — the Hugging Face callback measures wall-clock step time live; deterministic rules catch step-time cliffs and dataloader-bound runs, and watch --stall-timeout warns when the log stops growing. Stable rule IDs TP-DIVERGE , TP-DEAD-RUN , … documented in RULES.md https://github.com/Mormolykos/trainproof/blob/main/RULES.md , an honest PASS that lists which checks ran and which were skipped, and --json output built for CI pipelines and AI coding agents. A failure nobody injected. Injected faults are cleaner than real ones. That is the known limitation of any fault-injection study, and it is why this one does not stand alone. The gallery now ships examples/real world/xtts diverged : a 9.8-hour Coqui XTTS fine-tune that diverged on its own, in my own work, with nobody touching it. All three seeds of every configuration ship too — 18 runs at seeds 42/43/44, where before only one seed per configuration was committed — so "three seeds out of three" became a claim you can check rather than one you have to take on trust. Update: the linter failed its own test This article argues that a training run can look healthy while learning nothing. Auditing trainproof before a release, I found it doing exactly that — to itself. The question was mundane. What happens to a run whose loss is exactly 0.0 on every step? That is a real failure with a boring cause: if every target label is masked to -100 , cross-entropy returns zero and the model learns nothing at all. trainproof's answer was PASS . Every loss-shape check — flat curve, divergence, no-improvement — was guarded by a 0 test, put there to avoid dividing by zero. On an all-zero curve all three guards bailed out, silently. The run reached a passing verdict, and that verdict then named those same three checks as having run . That is worse than missing the failure. A tool that reports "I checked this" when it did not is not a weak alarm, it is a false one — and it is the same shape as the bad labels run above: something that looks like evidence of health and is nothing of the kind. The same audit found the hole in three more places, including a baseline comparison that rated a zero-loss run as favourable , because a loss floor of zero beats every baseline there is. v0.12.0 is the fix, and it encodes one sentence: a skipped check is not a passed check. A PASS now names every check that ran and every check that did not, each with its reason, as structured data rather than prose. Judging a log that has no evaluation set now says so out loud: Ran: dead-run, divergence, flat-loss, grad-spike, lr, zero-grad, zero-loss. Skipped: loader no loader time/step time pair in the log ; overfit no eval loss in the log - this run has no generalisation signal at all ; step-time no step time column in the log . You can read that and know what the pass is worth. Nothing about how trainproof judges a run changed — all 38 locked verdicts are byte-identical to the previous release. What changed is that it stopped overstating what it had looked at.