A clean run proves the process executed. It says nothing about what the pipeline learned, from which rows, in what state, or whether the saved result can be trusted anywhere else.
A model scores 0.83 in validation, the notebook runs top to bottom without a single error, and three weeks after deployment the predictions are useless. Nothing crashed at any point in that story. That is what makes AI workflow bugs different from ordinary Python bugs: the APIs happily accept code that violates a data, state, shape or artifact contract. The penalty then arrives as a believable number instead of a traceback. A clean run proves the process executed. It says nothing about what the pipeline learned, from which rows, in what state, or whether the saved result can be trusted anywhere else.
The seven mistakes below all share that silence, and each one comes with the check that catches it at the boundary where it starts.
| Stage | Silent Mistake | Misleading Symptom | The Check |
|---|---|---|---|
| Preprocessing | |||
| Transform fitted before the split | Validation score is optimistically inflated | Locate every fit call; name the rows visible at that moment | |
| Splitting | |||
| Related rows on both sides of the split | Strong validation, weak on new entities | Group- or time-aware splitter matched to the real boundary | |
| Serving | |||
| Second hand-written preprocessing path | Train and serve outputs drift apart silently | One fixture through both paths; assert outputs identical | |
| Randomness | |||
| One seed treated as reproducibility | Reruns differ despite the seeded library | Record seeds, data, code, config, and dependencies | |
| Evaluation | |||
eval() and no_grad() used interchangeably |
|||
| Dropout or batch norm active during validation | Both calls in the loop, then model.train() on resume |
||
| Loss Boundary | |||
Broadcast hides a [batch, 1] vs [batch] mismatch |
|||
| Plausible loss from the wrong computation | assert output.shape == target.shape before the loss |
||
| Artifact | |||
| Saved model treated as inert data | Code execution or version breakage on load | Trusted sources only; smoke-test in the serving environment |
1. Fitting Preprocessing Before Splitting the Data #
Here is a demonstration worth running once. Take 100 samples of pure random noise, 1,000 features wide, with labels assigned by coin flip. Now ask SelectKBest
for the 20 "best" features and cross-validate a classifier on the survivors:
sel = SelectKBest(f_classif, k=20).fit(X, y) # fit on ALL rows
scores = cross_val_score(model, sel.transform(X), y, cv=5)
On data with nothing in it to learn, that pair of lines still reports 0.83 accuracy. The selector saw every row, including the ones each fold later treats as unseen, so information from the held-out data already shaped the features being evaluated. Moving the selection inside a ** scikit-learn pipeline** so each fold fits its own transform drops the same experiment to 0.49, which is the honest answer for noise. That rule generalizes past feature selection to scaling, imputation, and dimensionality reduction. The diagnostic is a search rather than a rerun: find every
fit
and fit_transform
in the workflow, then name the rows that were visible at that moment. scikit-learn keeps a whole catalog of these pitfalls, with leakage at the top for a reason.
2. Randomly Splitting Rows That Are Not Independent #
A random split answers one question: whether the model can predict rows it has not seen. Production usually asks a harder one — whether it can predict users, patients, or devices it has not seen. When five rows belong to the same user, a random split scatters them across training and validation. The model then collects credit for recognizing users rather than generalizing to new ones. A synthetic version with 60 users and near-duplicate rows per user scores 0.97 under train_test_split
and drops to 0.89 the moment GroupShuffleSplit
keeps each user on one side of the line. That eight-point gap is the memorization being refunded. Grouped data wants GroupKFold
or GroupShuffleSplit
, while time-ordered data wants TimeSeriesSplit
, since a random split happily trains on the future to predict the past. Stratifying on the label does nothing here, and train_test_split
has no notion of groups at all. The cross-validation guide maps which splitter matches which boundary. Decide what the model must generalize beyond — an entity or a point in time — before choosing one.
3. Running Different Preprocessing Code at Training and Inference #
Skew looks like leakage's twin but points the other way. Leakage lets evaluation borrow from held-out rows, while skew applies a different transformation path after training. The skew version usually starts innocently, with a notebook that scaled features one way and a serving function that reimplements the "same" scaling by hand. The failure has real size. Re-learning a scaler on a five-row serving batch instead of reusing the fitted one can shift the very same fixture by almost four standard units — the difference between a prediction and a coin flip. Similar-looking code is not a contract. Inference has to use the exact learned parameters, feature order, dtype, and missing-value rules that training used. The cheapest guarantee is shipping the fitted pipeline object itself down both paths. Checking for it takes one raw fixture, pushed through both the training path and the serving path. If the two outputs differ anywhere — in names, order, dtype, shape, or values — the serving path is lying about something.
4. Seeding One Library and Calling the Experiment Reproducible #
random.seed(42)
at the top of a script mostly buys reassurance. Python's random
module, NumPy, and PyTorch each run their own generator, and seeding the first one leaves the other two producing exactly the unseeded output they would have produced anyway. A Data
with worker processes adds its own seeding rules on top.
Fully deterministic kernels have to be requested explicitly, sometimes at a performance cost, as the PyTorch reproducibility notes spell out. Those notes also set the honest ceiling. Identical results are not promised across PyTorch releases, platforms, or CPU and GPU execution, no matter how many seeds are set. Reproducibility is therefore a recording problem more than a seeding problem.
A run that logs its seeds, data snapshot, code version, configuration, and dependency versions can be reconstructed, while a lone 42 cannot rebuild an environment. The ** Weights & Biases crash course** shows one practical way to make that recording automatic.
5. Confusing Evaluation State with Disabled Gradients #
model.eval()
and torch.no_grad()
get treated as interchangeable because both appear in validation loops, but they control different machinery. Evaluation mode switches training-sensitive modules such as dropout and batch normalization into inference behavior, while no_grad
only stops autograd from recording work.
Run a dropout model twice on the same input under no_grad
while still in training mode and the two outputs differ, because dropout is still firing. In one small model the pair came back as -0.1410 and 0.0071. Switch to model.eval()
and the same two calls return one identical answer.
The dependency runs the other way too, since a model in eval mode without no_grad
still records gradients on every forward pass. A validation loop needs both switches, and the model set back to training mode afterward:
model.eval()
with torch.no_grad():
val_loss = criterion(model(x_val), y_val)
model.train()
The autograd notes cover the boundary in detail. When nothing inside the block will ever need gradients, torch.inference_mode()
locks that door harder than no_grad
does. Even a model that currently lacks dropout deserves the explicit eval()
call, because architectures change and the call costs nothing.
6. Letting Broadcasting Hide a Wrong Tensor Shape #
Broadcasting is a feature until it reaches a loss function. Suppose the prediction comes out shaped [batch, 1]
while the target is [batch]
. Inside MSELoss
the subtraction broadcasts that pair into a full batch-by-batch matrix, so every prediction gets compared against every label.
With a batch of 32 that means a 32-by-32 grid, and the loss computes anyway: 1.63, where the correctly shaped version gives 1.85 on the same tensors. Nothing about 1.63 looks suspicious, and no exception ever fires. PyTorch does emit a UserWarning
here, which tests should promote to an error. ** MSELoss** documents the target as matching the input's shape, so the correction is deciding the contract once and enforcing it at the boundary:
pred = model(x).squeeze(1) # [batch, 1] -> [batch], on purpose
assert pred.shape == target.shape
loss = criterion(pred, target)
Not every broadcast is a bug, as the broadcasting semantics make clear. The mistake is allowing implicit expansion at a boundary where the loss, metric, or label contract demands exact agreement.
7. Treating a Saved Model Like an Inert, Portable File #
The last boundary is the file itself. A pickled model — whether written by pickle
, joblib
, or cloudpickle
— is not passive data. one can execute arbitrary code, and a five-line file with a malicious __reduce__
method will happily run its payload during pickle.load
without raising anything. So an artifact from a source nobody has verified should simply never be loaded.
Version drift causes less drama but bites the same workflow, because ** scikit-learn does not support** a model saved under a different library version. Ship every artifact with its training recipe, data reference, dependency versions, and the validation score it claims.
Before promotion, load it in the real serving environment and push a fixed fixture through the complete preprocessing-and-prediction path. Alternative formats move these problems around rather than deleting them, so no single format is the safety fix.
Making the Workflow Prove Its Boundaries #
None of these seven mistakes announces itself, which is why the review has to be a habit rather than a reaction. Four questions cover the territory. What did each step learn, and from which rows? What code converts raw input at serving time, and is it the contract training used? What state and shape reached the metric? And which environment is trusted to load the artifact? A workflow that answers those from code and recorded metadata has earned its score. One that cannot is holding a promising guess.
is a software developer and tech writer. Before devoting her work full time to technical writing, she managed—among other intriguing things—to serve as a lead programmer at an Inc. 5,000 experiential branding organization whose clients include Samsung, Time Warner, Netflix, and Sony.