# Loss Curves Lie: Building a Deterministic Linter for ML Training Runs > Source: > Published: 2026-08-14 07:35:46+00:00 # Loss Curves Lie: Building a Deterministic Linter for ML Training Runs *A run learning pure noise reduced its loss by 62% and looked textbook-healthy. trainproof catches the training failures that can be caught — and is honest about the ones that can't.* ## trainproof v0.18.0activeDeterministic PASS/WARN/FAIL verdicts across a training run's whole life — before it starts (dataset, tokenizer, and an environment preflight that catches a stack which will not import, an unreadable checkpoint, or too little system RAM), a live guardian during training, a post-run autopsy after, and a baseline compare. Reads five log formats including TensorBoard event files, decoded from the wire format with no torch, tensorflow or tensorboard installed. No ML judging ML: every verdict is a rule that fires or doesn't, every finding cites its exact numbers, and a check that could not run is reported as NOT-CHECKED rather than as a pass. - Objective checks: catches a cross-entropy ignore_index that is also a valid output class - a collision that silently deletes a target from the loss while leaving the loss curve indistinguishable from a correct run (0.0035 vs 0.0034) - trainproof doctor . — zero-config autopsy: discovers every training log in a directory, judges them, prints a triage-sorted summary - Live guardian: a one-line Hugging Face callback; opt-in stop_on_fail aborted a diverging Qwen2.5-3B QLoRA at step 20 of 300 in a real demo - Preflight: lints dataset + tokenizer before a single GPU-second (malformed JSONL, empty rows, duplicates, missing eos/pad, over-length) - compare catches shuffled-labels corruption that single-run loss rules cannot — the relative loss floor sits ~6× a known-good baseline - Stable rule IDs (TP-DIVERGE, TP-OVERFIT, TP-DEAD-RUN, …) documented in RULES.md, plus --json output built for CI and AI coding agents - CONTRACTS.md pins the promises — exit codes, JSON schema policy, rule-ID stability and a verdict-stability guarantee; SARIF 2.1.0 output turns findings into GitHub PR annotations - A PASS names every check that ran and every check it skipped, with the reason — a skipped check is never reported as a passed one - Grounded in an 18-run fault-injection study (Qwen2.5-3B QLoRA, RTX 5080) — six configurations at three seeds each, every real log shipped in-repo — plus a 9.8-hour Coqui XTTS fine-tune that diverged on its own, the first shipped failure nobody injected - Reads five log formats including TensorBoard event files, decoded from the wire format with no tensorflow, tensorboard, protobuf, torch or numpy — validated byte-exact against tensorboard's own EventAccumulator, and the rules are exercised on real runs from three frameworks: HuggingFace, Coqui XTTS and PyTorch Lightning ### Recent Releases **v0.18.0** 2026-08-09 — v0.18.0 - the objective release **v0.17.0** 2026-08-02 — v0.17.0 — the lint gate, and one log that used to vanish **v0.16.0** 2026-08-02 — v0.16.0 — the rule registry (no behaviour change) **v0.15.0** 2026-08-01 — v0.15.0 — the before-the-GPU release I kept losing GPU hours to runs that were already dead. A fine-tune that had quietly gone to NaN somewhere after step 300 and trained on garbage the rest of the night. A run that was training at learning-rate-*zero* the whole time and updating nothing. A dataset with a few hundred broken rows I only discovered at the end. Every time, the pattern was the same: the run *looked* like it was working, and nothing told me otherwise until the hours were already spent. So I built a linter for training runs — **trainproof** — and then I spent days trying to prove it wrong. ``` pip install trainproof trainproof doctor . # zero-config: discovers and judges every training log it finds ``` *Repo: github.com/Mormolykos/trainproof (MIT)* ## The philosophy: no ML judging ML The most important decision in trainproof is what it *isn't*. There's no model scoring your run. No "87%-confidence this looks unhealthy." **Every verdict is a deterministic rule that either fires or it doesn't, and every finding cites the exact numbers behind it.** When it can't be sure, it says so instead of guessing. That constraint is the whole point. A reliability tool that hallucinates is worse than no tool, because now you don't trust your own alarms. trainproof's engine is IO-free and doesn't even import torch — it reads plain logs and applies rules. You can read every rule and know exactly why it fired. ## Validating the rules: controlled fault injection You validate a detector by feeding it faults whose answer you already know, then measuring what it catches. One base setup — a Qwen2.5-3B QLoRA on an RTX 5080 — run six ways: **healthy**— the control.** lr_hot**— learning rate cranked 100× too high.** lr_zero**— learning rate at zero.** fp16_nan**— an fp16 overflow that NaNs the loss.** bad_labels**— the dataset's labels shuffled into pure noise.** overfit**— 64 training samples and many epochs: pure memorisation, with a held-out eval set to prove it. Three seeds each: **18 runs**, real logs, all shipped in the repo so you can reproduce every verdict. A nineteenth log ships beside them with no injected fault at all — a 9.8-hour Coqui XTTS fine-tune that diverged on its own. Four of the five failures got caught fast. `lr_hot` spiked the gradient norm to **2,650× the median** — flagged in seconds. The NaN and the flatline were trivial. But one config beat the tool completely, and it's the one that taught me the most. ## The one that fooled it: shuffled labels `bad_labels` was pure garbage — a dataset that *cannot be learned*, because the labels no longer correspond to the inputs. And that run **reduced its loss by 62%.** On its own loss curve it looked like textbook-healthy training: a clean downward slope. It was learning absolutely nothing useful — just memorizing the statistics of noise, which any sufficiently large network will happily do. From a single run's loss curve, **it is indistinguishable from a real run.** No loss-only rule, looking at that run in isolation, can catch it. That's not a gap I papered over — I wrote it straight into the README. It's also *why* trainproof has a `compare` mode: the failure becomes obvious the moment you put the run next to a known-good baseline and look at the *relative* floor it reaches. The corrupted run's loss floor sits in a different regime. One run in isolation lies; two runs side by side tell the truth. ## What trainproof actually does It covers a run's whole life, and every stage exits with a status code so it drops straight into CI: **Before a single GPU-second —** Lints the dataset and tokenizer: malformed JSONL (with the line number), empty rows, duplicates, missing`preflight` .`eos_token` ,`pad == eos` , over-length samples. Catch the broken dataset*before*you rent the GPU.**During training — the guardian.** A one-line Hugging Face callback. It warns by default; flip on`stop_on_fail` and it will abort a doomed run itself. In one guarded demo, it killed a diverging run at**step 20 of 300 scheduled steps**— so 280/300 (≈93%) of the scheduled steps never executed. (That's a statement about*that run*, not a promise about your GPU bill — but a diverging run you stop at step 20 is a diverging run you didn't pay to finish.)**After —** Read the finished log and classify it: diverged, flatlined, NaN'd, spiked, overfitting.`epoch` and`doctor` .**Versus a baseline —** The relative-floor rules that catch the shuffled-labels case.`compare` . It reads the logs you already produce — Hugging Face `trainer_state.json` , Coqui, plain JSONL/CSV — so there's nothing to instrument. ## Try it ``` pip install trainproof ``` - Repo (with all 18 fault-injection logs, plus the one that broke by itself): [https://github.com/Mormolykos/trainproof](https://github.com/Mormolykos/trainproof) - MIT licensed. If you fine-tune, you've had at least one of these failures. I'd genuinely like to know **which failure mode has burned your GPU hours** — if a deterministic check would have caught it, tell me on the repo and it goes in, with credit. ## Update: what came after Since this article was written, trainproof has grown from the experiment above into a fuller tool. The release timeline at the top of this page is generated from GitHub Releases and PyPI, so it is always current — this section covers what changed in kind, not which version you are on: — point it at a directory; it discovers every training log, judges them all, and prints a triage-sorted summary. The six configurations above are now one command.`trainproof doctor .` **Guardian telemetry**— the Hugging Face callback measures wall-clock step time live; deterministic rules catch step-time cliffs and dataloader-bound runs, and`watch --stall-timeout` warns when the log stops growing.**Stable rule IDs**(`TP-DIVERGE` ,`TP-DEAD-RUN` , …) documented in[RULES.md](https://github.com/Mormolykos/trainproof/blob/main/RULES.md), an honest PASS that lists which checks ran and which were skipped, and`--json` output built for CI pipelines and AI coding agents.**A failure nobody injected.** Injected faults are cleaner than real ones. That is the known limitation of any fault-injection study, and it is why this one does not stand alone. The gallery now ships`examples/real_world/xtts_diverged` : a 9.8-hour Coqui XTTS fine-tune that diverged on its own, in my own work, with nobody touching it. All three seeds of every configuration ship too — 18 runs at seeds 42/43/44, where before only one seed per configuration was committed — so "three seeds out of three" became a claim you can check rather than one you have to take on trust. ## Update: the linter failed its own test This article argues that a training run can look healthy while learning nothing. Auditing trainproof before a release, I found it doing exactly that — to itself. The question was mundane. What happens to a run whose loss is *exactly* 0.0 on every step? That is a real failure with a boring cause: if every target label is masked to `-100` , cross-entropy returns zero and the model learns nothing at all. trainproof's answer was **PASS**. Every loss-shape check — flat curve, divergence, no-improvement — was guarded by a `> 0` test, put there to avoid dividing by zero. On an all-zero curve all three guards bailed out, silently. The run reached a passing verdict, and that verdict then *named those same three checks as having run*. That is worse than missing the failure. A tool that reports "I checked this" when it did not is not a weak alarm, it is a false one — and it is the same shape as the `bad_labels` run above: something that looks like evidence of health and is nothing of the kind. The same audit found the hole in three more places, including a baseline comparison that rated a zero-loss run as *favourable*, because a loss floor of zero beats every baseline there is. **v0.12.0** is the fix, and it encodes one sentence: *a skipped check is not a passed check.* A PASS now names every check that ran and every check that did not, each with its reason, as structured data rather than prose. Judging a log that has no evaluation set now says so out loud: ``` Ran: dead-run, divergence, flat-loss, grad-spike, lr, zero-grad, zero-loss. Skipped: loader (no loader_time/step_time pair in the log); overfit (no eval_loss in the log - this run has no generalisation signal at all); step-time (no step_time column in the log). ``` You can read that and know what the pass is worth. Nothing about how trainproof judges a run changed — all 38 locked verdicts are byte-identical to the previous release. What changed is that it stopped overstating what it had looked at.