{"slug": "training-validation-and-test-sets-in-machine-learning-a-practical-guide", "title": "Training, Validation, and Test Sets in Machine Learning: A Practical Guide", "summary": "A developer's practical guide explains why models that score 97% in training often underperform in production, and lays out the standard three-way split of training, validation, and test sets to measure generalization honestly. Using a 10,000-email spam classifier as an example, the guide recommends a 70/15/15 split with stratified sampling, selecting between logistic regression and random forest on the validation set, and reporting the single final score on the held-out test set. It also warns that repeatedly tuning against validation data gradually biases that estimate, so only the untouched test set gives a trustworthy read on real-world performance.", "body_md": "You train a model, check the accuracy, and get 97%. Looks great. You deploy it, and it performs much worse on real data.\n\nThis is not unusual. The model learned the training data well, but the evaluation did not tell you what actually matters how it performs on data it has never seen before.\n\nThat is what dataset splitting is for.\n\nWhen a model trains on a dataset, it adjusts its parameters to fit those examples. If you then evaluate it on the same data, you are asking it to repeat something it already learned from. The score will look good, but it tells you almost nothing about real-world performance.\n\nWhat you actually care about is generalization whether the model can make correct predictions on new examples it was not trained on. To measure that honestly, you hold some data back and keep it separate during training.\n\nThat is the core idea. But you typically need three separate portions, not two. Here is why.\n\nThe training set is what the model learns from. It sees these examples, computes how wrong its predictions are, and adjusts its parameters to reduce that error over many iterations.\n\nIf you are building a model to predict house prices, the training set would contain houses with features like size, location, and number of bedrooms, along with their actual prices. The model uses these to learn the relationship between the features and the target.\n\nEverything the model knows comes from this data.\n\nAfter training, you need to make decisions: which algorithm to use, how many layers to include, what regularization strength to apply. These choices are called hyperparameters, and they are not learned from data you set them yourself.\n\nYou cannot make these decisions reliably by looking at training performance, because the model has already seen that data and tends to score well on it. So you use a separate validation set.\n\nThe validation set lets you compare different models or configurations and pick the one that performs best on data the model has not trained on. You can check validation scores as many times as you want during development. That is exactly what it is for.\n\nOne thing to keep in mind: because you are using validation scores to guide your decisions, your choices are gradually tuned toward that particular validation data. This means validation performance is not a fully neutral measure of how the model will do in production. It is still far more reliable than training performance, but it is not the final word.\n\nThe test set is held back until development is completely finished. You use it once after you have selected a model, finalized your hyperparameters, and made no further changes.\n\nThis is what gives you a trustworthy estimate of real-world performance. Because no model decisions were based on this data, the score reflects how the model will likely behave on genuinely new examples.\n\nSay you are building a spam classifier with 10,000 labeled emails.\n\nYou split it like this:\n\nFor a classification problem like this, using stratified splitting is a good idea. It ensures that the proportion of spam and non-spam emails is roughly the same in each split, rather than ending up by chance with most spam examples in training and very few in the test set.\n\nYou train a logistic regression model on the 7,000 training emails and evaluate it on the validation set. Then you try a random forest and compare. You go with whichever does better on validation. After picking the better model, you finalize it no more changes.\n\nThen you run it once on the 1,500 test emails. That score is what you report.\n\nHere is what the core workflow looks like in code. `X` is a numerical feature matrix and `y` contains the labels, both already prepared before this step. If you are working with raw email text, you would typically convert it to numerical features first for example using `TfidfVectorizer`. In that case, fit the vectorizer on the training text only, then use it to transform the validation and test text. Fitting it on the full dataset before splitting leaks information from the test set into your features.\n\n``` python\nfrom sklearn.model_selection import train_test_split\nfrom sklearn.linear_model import LogisticRegression\nfrom sklearn.ensemble import RandomForestClassifier\nfrom sklearn.metrics import accuracy_score\n\n# X: numerical feature matrix, y: labels (0 = not spam, 1 = spam)\n\n# Step 1: Set aside the test set first\nX_temp, X_test, y_temp, y_test = train_test_split(\n    X, y, test_size=0.15, stratify=y, random_state=42\n)\n\n# Step 2: Split remaining data into training and validation\nX_train, X_val, y_train, y_val = train_test_split(\n    X_temp, y_temp, test_size=0.176, stratify=y_temp, random_state=42\n)\n# 0.176 of 85% gives roughly 15% of the original data for validation\n\n# Step 3: Train two models and compare on validation\nlr = LogisticRegression(max_iter=1000)\nlr.fit(X_train, y_train)\nlr_val_acc = accuracy_score(y_val, lr.predict(X_val))\n\nrf = RandomForestClassifier(n_estimators=100, random_state=42)\nrf.fit(X_train, y_train)\nrf_val_acc = accuracy_score(y_val, rf.predict(X_val))\n\nprint(f\"Logistic Regression validation accuracy: {lr_val_acc:.3f}\")\nprint(f\"Random Forest validation accuracy:       {rf_val_acc:.3f}\")\n\n# Step 4: Select the better model based on validation results\n# Suppose Random Forest performed better -- pick it as the final model\nfinal_model = rf\n\n# Step 5: Evaluate the final model on the test set -- do this once only\ntest_acc = accuracy_score(y_test, final_model.predict(X_test))\nprint(f\"Test accuracy: {test_acc:.3f}\")\n```\n\nThe numbers printed above are hypothetical -- this code has not been run on an actual dataset. The point is to show the structure: validation scores guide the choice between models, and the test set is only used at the very end.\n\nOne note on metrics: accuracy works well when spam and non-spam examples are roughly balanced. If your dataset has far more non-spam than spam, a model that labels everything as non-spam can look accurate while being useless. In those cases, also look at precision, recall, and F1-score to get a clearer picture of how the model actually performs.\n\nOne more thing worth noting: if your model needs preprocessing like scaling features or filling in missing values fit that transformation only on training data. Then apply the same fitted transformation to the validation and test sets. If you fit preprocessing on the full dataset before splitting, statistics from the test set can leak into training, which makes evaluation less trustworthy.\n\nUsing the test set once to evaluate a finalized model is fine. That is the whole point.\n\nThe problem happens when you use the test score to make decisions. You check the test score, change your model, check again, change something else, check again. Over many iterations, your choices become shaped by how those choices perform on the test set even though you never explicitly trained on it.\n\nThe result is a form of selection bias. You have been, in effect, selecting models and settings that happen to work better on your specific test data. The final test score ends up more optimistic than it should be.\n\nThis is not the same as overfitting, where a model memorizes training examples. Overfitting happens to the model. Repeated test-set use is a problem with the evaluation process the developer's decisions start to reflect patterns in the test data even when the model itself has never seen it.\n\nThe fix is straightforward: make all development decisions using the validation set. Reserve the test set for one final evaluation after everything is finalized.\n\n**Evaluating on training data.** A model almost always scores higher on data it trained on. This does not reflect generalization. Always use held-out data to evaluate.\n\n**Using test scores to guide decisions.** Once your choices are influenced by test results, the test set no longer provides an independent estimate. This is the scenario described above.\n\n**Data leakage.** This happens when information that should not be available at prediction time influences training or evaluation. A common case: a feature derived from the target variable ends up in the training data. Another: a model trained on historical data accidentally includes future information. The model learns to use signals that will not exist in production, which inflates evaluation scores.\n\nLeakage is different from repeated test-set use. Leakage contaminates the data itself. Repeated test-set use contaminates the evaluation process.\n\n**Overlap between splits.** If similar or duplicate examples appear in both training and test data, the evaluation is no longer measuring genuine generalization. This is easy to overlook when a dataset has near-duplicate records or when related examples (like multiple photos of the same person) are split randomly.\n\n**Random splits on time-based data.** If your data has a time dimension and you are predicting future events, a random split mixes future information into the training set. The model effectively learns from data it should not have access to at prediction time. For time-series problems, use earlier data for training and later data for evaluation.\n\n**Assuming one ratio works for every dataset.** A 70/15/15 split is a reasonable starting point, but it is not a universal rule. Smaller datasets may need more data in training. If you are using cross-validation, the split strategy looks different. Match the approach to your specific situation.\n\nA high accuracy score only means something if the evaluation process was set up correctly.\n\nTraining, validation, and test sets each have a distinct role. Mixing them up, or leaking test information into development decisions, can make a model look better than it is. Getting the split right is one of the more straightforward things you can do to make your results trustworthy.", "url": "https://wpnews.pro/news/training-validation-and-test-sets-in-machine-learning-a-practical-guide", "canonical_source": "https://dev.to/aditya_d_sharma/training-validation-and-test-sets-in-machine-learning-a-practical-guide-cii", "published_at": "2026-10-18 18:11:09+00:00", "updated_at": "2026-10-18 18:17:33.756769+00:00", "lang": "en", "topics": ["machine-learning", "ai-research"], "entities": [], "also_reported_by": [], "alternates": {"html": "https://wpnews.pro/news/training-validation-and-test-sets-in-machine-learning-a-practical-guide", "markdown": "https://wpnews.pro/news/training-validation-and-test-sets-in-machine-learning-a-practical-guide.md", "text": "https://wpnews.pro/news/training-validation-and-test-sets-in-machine-learning-a-practical-guide.txt", "jsonld": "https://wpnews.pro/news/training-validation-and-test-sets-in-machine-learning-a-practical-guide.jsonld"}}