{"slug": "show-hn-trainlint-static-checks-for-ml-bugs-that-don-t-crash", "title": "Show HN: Trainlint, static checks for ML bugs that don't crash", "summary": "A developer released Trainlint, an offline static checker that flags 14 common machine-learning bugs that do not crash training, installable via `uvx trainlint check .` or `pip install trainlint` on Python 3.10 or newer with no dependencies. The tool parses code with Python's `ast` module and never runs it, covering rules TL001 through TL014 — including scalers fitted before a split, `model.eval()` inside a training loop without `model.train()`, and `softmax` at the end of a model trained with a logits loss — and ships four Agent Skills that `trainlint skills install` copies into `./.claude/skills` to teach coding agents to avoid the same mistakes.", "body_md": "Agent skills and an offline checker for ML bugs that do not crash.\n\nThe 5,000 features and the labels are random, so no model can beat 50%. Picking features before\ncross-validating reports 85%. The GIF is rendered from the real output of the commands shown;\n`assets/make_assets.py` rebuilds it.\n\n```\nuvx trainlint check .\n```\n\nor `pipx install trainlint` / `pip install trainlint`. It needs Python 3.10 or newer, has no\ndependencies, and never touches the network or a GPU.\n\n```\nuvx trainlint check train.py     # find the bugs in a script, a folder, or a repo\nuvx trainlint skills install     # teach your coding agent to avoid them\n```\n\n`skills install` copies four skills into `./.claude/skills`. Use `--agent agents` for\n`./.agents/skills`, `--global` for your user folder, or `--dest DIR` for any other place.\n\nMore ways to run the checker:\n\n```\ntrainlint check src/ --ignore TL005   # skip a rule\ntrainlint check . --format json       # for scripts and CI\ntrainlint rules                       # list every rule\n```\n\nExit code 0 means no findings, 1 means findings, 2 means a usage error. To silence a finding on one line:\n\n```\nscaler.fit(X_all)  # trainlint: ignore[TL002] fitting on the pooled set is intended here\n```\n\nIn GitHub Actions:\n\n```\n- run: uvx trainlint check .\n```\n\nML code fails quietly. A scaler fitted before the split, a model left in eval mode, or a softmax\nin front of `CrossEntropyLoss` still trains and still prints a number. Coding agents and people\nboth write these mistakes, and general linters do not know what the code is for. trainlint is a\nsmall offline checker for the common cases, plus skills that tell an agent how to avoid them in\nthe first place.\n\n- Fourteen rules, TL001 to TL014, listed below and explained with examples in\n[docs/rules.md](https://github.com/Satvik-jain/trainlint/blob/main/docs/rules.md) .\n- Each finding shows the line, a pointer into the code, and a one-line fix.\n- Pure standard library. It parses the code and never runs it.\n- Text or JSON output, rule selection, and per-line ignore comments.\n- Four [Agent Skills](https://agentskills.io/specification) that cover the same ground and run\nthe checker when it is installed.\n\n| Rule | Flags | \n|---|---|\n| TL001 | Scaler, encoder, selector or resampler fitted on all rows before a split or cross-validation | \n| TL002 | Transform or model fitted on a validation or test split | \n| TL003 | `model.eval()` inside a training loop with no`model.train()` to switch back | \n| TL004 | Evaluation under `no_grad` in a script that never calls`model.eval()` | \n| TL005 | Training script that never sets a random seed | \n| TL006 | float16 autocast without a `GradScaler` , or a scaled loss stepped without`scaler.step` | \n| TL007 | Script that resumes from checkpoints but saves no optimizer state | \n| TL008 | `softmax` or`sigmoid` at the end of a model trained with a logits loss | \n| TL009 | Gradients never cleared, or cleared between `backward()` and`step()` | \n| TL010 | Loss added to a running total or list as a tensor, keeping every graph alive | \n| TL011 | `y_true` and`y_pred` swapped, or class labels passed to`roc_auc_score` | \n| TL012 | Validation or test `DataLoader` with`drop_last=True` | \n| TL013 | Data rescaled with its own mean, std, min or max before a split | \n| TL014 | Training `DataLoader` with no`shuffle` ,`sampler` or`batch_sampler` | \n\n| Skill | Covers | \n|---|---|\n| `ml-data-leakage` | Splits, pipelines, grouped and time-ordered data, resampling, the test set | \n| `ml-training-loop` | Modes, gradient order, loss inputs, shapes, seeds, mixed precision | \n| `ml-metrics-eval` | Argument order, scores versus labels, averaging, imbalance, variance | \n| `ml-checkpoint-resume` | What a checkpoint must hold, best versus last, verifying a resume | \n\nThe checks parse each file with Python's `ast` module and match calls by name. There is no type\ninference and no cross-file analysis, so:\n\n- Splits are recognized by variable names such as `X_train` ,`val_loader` and`y_test` .\n- A bug whose cause sits in another file is not seen.\n- A heuristic can be wrong. When it is, add an ignore comment and open an issue.\n\n`python -m bench` runs the checker over a corpus of 20 buggy scripts, one or two per rule, each\nwith a fixed twin:\n\n```\nrule    buggy files  caught   false alarms on fixed twins\nTL001             3       3                             0\nTL002             1       1                             0\nTL003             1       1                             0\nTL004             1       1                             0\nTL005             1       1                             0\nTL006             2       2                             0\nTL007             1       1                             0\nTL008             2       2                             0\nTL009             2       2                             0\nTL010             1       1                             0\nTL011             2       2                             0\nTL012             1       1                             0\nTL013             1       1                             0\nTL014             1       1                             0\ntotal            20      20                             0\n```\n\nI wrote that corpus together with the rules, so it guards against regressions and says nothing about accuracy on code I have not seen.\n\nFor a check on someone else's code, I ran it over all 93 Python files of\n[pytorch/examples](https://github.com/pytorch/examples) at commit `acc295d` and read every\nfinding:\n\n```\ngit clone https://github.com/pytorch/examples && cd examples\ngit checkout acc295dc7b90714f1bf47f06004fc19a7fe235c4\ntrainlint check .\n```\n\nIt reports 22 findings. By my reading, 20 are real: 15 scripts never set a seed (one declares a\n`--seed` option and never uses it), 3 tensor-parallel scripts never clear gradients, one script\nresumes from a snapshot without saving optimizer state, and one trains on a loader that never\nshuffles. Two more are correct as written but harmless, because those models have no dropout or\nbatch norm to leave in training mode. Earlier runs also flagged three distributed-autograd scripts\nfor missing `zero_grad()` and one torchtext iterator for not shuffling. Both were false positives\nand are now fixed. This is one repository of tutorial code and I did the reviewing, so\ntreat it as a spot check. It tells you nothing about what the rules miss.\n\n| Tool | What it covers | Where trainlint differs | \n|---|---|---|\n| ruff, pylint, flake8 | General Python style and bugs | None of their rules know about splits, eval mode or metrics. Use them alongside trainlint. | \n| [TorchFix](https://github.com/meta-pytorch/torchfix) | Removed and deprecated PyTorch functions, `requires_grad` typos, unsafe`torch.load` | Its README lists no rules for training logic. trainlint does not check API deprecations. | \n| [mllint](https://github.com/bvobart/mllint) | Repository-level quality of ML projects: version control, dependencies, code quality | It looks at project hygiene. trainlint reads training and evaluation code. | \n| [Deepchecks](https://github.com/deepchecks/deepchecks) | Checks that run on real datasets and models, including drift | Deepchecks needs your data and model loaded. trainlint reads source only and runs nothing. | \n| General skill collections | Engineering process for coding agents | They are not specific to ML. trainlint's skills target training and evaluation code and come with a checker. | \n\n- Jupyter notebook support\n- SARIF output for GitHub code scanning\n- Settings in `pyproject.toml`\n- More rules: `retain_graph` in a loop\n- A benchmark that measures how often an agent writes these bugs with and without the skills (needs an API key, so it will be optional)\n\nIssues labeled [good first issue](https://github.com/Satvik-jain/trainlint/labels/good%20first%20issue)\nare small and have a clear definition of done.\n\nFalse positives and missed bugs are the most useful reports. See\n[CONTRIBUTING.md](https://github.com/Satvik-jain/trainlint/blob/main/CONTRIBUTING.md) for setup and for how to add a rule.\n\n```\nuv sync\nuv run pytest\nuv run ruff check . && uv run ruff format --check .\nuv run python -m bench\n```\n\nMIT\n\nIf trainlint saved you an afternoon, a star on the repository helps other people find it.", "url": "https://wpnews.pro/news/show-hn-trainlint-static-checks-for-ml-bugs-that-don-t-crash", "canonical_source": "https://github.com/Satvik-jain/trainlint", "published_at": "2026-11-04 11:16:42+00:00", "updated_at": "2026-11-04 11:24:43.931678+00:00", "lang": "en", "topics": ["machine-learning", "ai-tools", "developer-tools", "ai-agents", "mlops"], "entities": ["Trainlint", "GitHub Actions", "Python", "Claude", "Satvik-jain"], "also_reported_by": [], "alternates": {"html": "https://wpnews.pro/news/show-hn-trainlint-static-checks-for-ml-bugs-that-don-t-crash", "markdown": "https://wpnews.pro/news/show-hn-trainlint-static-checks-for-ml-bugs-that-don-t-crash.md", "text": "https://wpnews.pro/news/show-hn-trainlint-static-checks-for-ml-bugs-that-don-t-crash.txt", "jsonld": "https://wpnews.pro/news/show-hn-trainlint-static-checks-for-ml-bugs-that-don-t-crash.jsonld"}}