You train a model, check the accuracy, and get 97%. Looks great. You deploy it, and it performs much worse on real data.
This is not unusual. The model learned the training data well, but the evaluation did not tell you what actually matters how it performs on data it has never seen before.
That is what dataset splitting is for.
When a model trains on a dataset, it adjusts its parameters to fit those examples. If you then evaluate it on the same data, you are asking it to repeat something it already learned from. The score will look good, but it tells you almost nothing about real-world performance.
What you actually care about is generalization whether the model can make correct predictions on new examples it was not trained on. To measure that honestly, you hold some data back and keep it separate during training.
That is the core idea. But you typically need three separate portions, not two. Here is why.
The training set is what the model learns from. It sees these examples, computes how wrong its predictions are, and adjusts its parameters to reduce that error over many iterations.
If you are building a model to predict house prices, the training set would contain houses with features like size, location, and number of bedrooms, along with their actual prices. The model uses these to learn the relationship between the features and the target.
Everything the model knows comes from this data.
After training, you need to make decisions: which algorithm to use, how many layers to include, what regularization strength to apply. These choices are called hyperparameters, and they are not learned from data you set them yourself.
You cannot make these decisions reliably by looking at training performance, because the model has already seen that data and tends to score well on it. So you use a separate validation set.
The validation set lets you compare different models or configurations and pick the one that performs best on data the model has not trained on. You can check validation scores as many times as you want during development. That is exactly what it is for.
One thing to keep in mind: because you are using validation scores to guide your decisions, your choices are gradually tuned toward that particular validation data. This means validation performance is not a fully neutral measure of how the model will do in production. It is still far more reliable than training performance, but it is not the final word.
The test set is held back until development is completely finished. You use it once after you have selected a model, finalized your hyperparameters, and made no further changes.
This is what gives you a trustworthy estimate of real-world performance. Because no model decisions were based on this data, the score reflects how the model will likely behave on genuinely new examples.
Say you are building a spam classifier with 10,000 labeled emails.
You split it like this:
For a classification problem like this, using stratified splitting is a good idea. It ensures that the proportion of spam and non-spam emails is roughly the same in each split, rather than ending up by chance with most spam examples in training and very few in the test set.
You train a logistic regression model on the 7,000 training emails and evaluate it on the validation set. Then you try a random forest and compare. You go with whichever does better on validation. After picking the better model, you finalize it no more changes.
Then you run it once on the 1,500 test emails. That score is what you report.
Here is what the core workflow looks like in code. X is a numerical feature matrix and y contains the labels, both already prepared before this step. If you are working with raw email text, you would typically convert it to numerical features first for example using TfidfVectorizer. In that case, fit the vectorizer on the training text only, then use it to transform the validation and test text. Fitting it on the full dataset before splitting leaks information from the test set into your features.
from sklearn.model_selection import train_test_split
from sklearn.linear_model import LogisticRegression
from sklearn.ensemble import RandomForestClassifier
from sklearn.metrics import accuracy_score
X_temp, X_test, y_temp, y_test = train_test_split(
X, y, test_size=0.15, stratify=y, random_state=42
)
X_train, X_val, y_train, y_val = train_test_split(
X_temp, y_temp, test_size=0.176, stratify=y_temp, random_state=42
)
lr = LogisticRegression(max_iter=1000)
lr.fit(X_train, y_train)
lr_val_acc = accuracy_score(y_val, lr.predict(X_val))
rf = RandomForestClassifier(n_estimators=100, random_state=42)
rf.fit(X_train, y_train)
rf_val_acc = accuracy_score(y_val, rf.predict(X_val))
print(f"Logistic Regression validation accuracy: {lr_val_acc:.3f}")
print(f"Random Forest validation accuracy: {rf_val_acc:.3f}")
final_model = rf
test_acc = accuracy_score(y_test, final_model.predict(X_test))
print(f"Test accuracy: {test_acc:.3f}")
The numbers printed above are hypothetical -- this code has not been run on an actual dataset. The point is to show the structure: validation scores guide the choice between models, and the test set is only used at the very end.
One note on metrics: accuracy works well when spam and non-spam examples are roughly balanced. If your dataset has far more non-spam than spam, a model that labels everything as non-spam can look accurate while being useless. In those cases, also look at precision, recall, and F1-score to get a clearer picture of how the model actually performs.
One more thing worth noting: if your model needs preprocessing like scaling features or filling in missing values fit that transformation only on training data. Then apply the same fitted transformation to the validation and test sets. If you fit preprocessing on the full dataset before splitting, statistics from the test set can leak into training, which makes evaluation less trustworthy.
Using the test set once to evaluate a finalized model is fine. That is the whole point.
The problem happens when you use the test score to make decisions. You check the test score, change your model, check again, change something else, check again. Over many iterations, your choices become shaped by how those choices perform on the test set even though you never explicitly trained on it.
The result is a form of selection bias. You have been, in effect, selecting models and settings that happen to work better on your specific test data. The final test score ends up more optimistic than it should be.
This is not the same as overfitting, where a model memorizes training examples. Overfitting happens to the model. Repeated test-set use is a problem with the evaluation process the developer's decisions start to reflect patterns in the test data even when the model itself has never seen it.
The fix is straightforward: make all development decisions using the validation set. Reserve the test set for one final evaluation after everything is finalized.
Evaluating on training data. A model almost always scores higher on data it trained on. This does not reflect generalization. Always use held-out data to evaluate.
Using test scores to guide decisions. Once your choices are influenced by test results, the test set no longer provides an independent estimate. This is the scenario described above.
Data leakage. This happens when information that should not be available at prediction time influences training or evaluation. A common case: a feature derived from the target variable ends up in the training data. Another: a model trained on historical data accidentally includes future information. The model learns to use signals that will not exist in production, which inflates evaluation scores.
Leakage is different from repeated test-set use. Leakage contaminates the data itself. Repeated test-set use contaminates the evaluation process.
Overlap between splits. If similar or duplicate examples appear in both training and test data, the evaluation is no longer measuring genuine generalization. This is easy to overlook when a dataset has near-duplicate records or when related examples (like multiple photos of the same person) are split randomly.
Random splits on time-based data. If your data has a time dimension and you are predicting future events, a random split mixes future information into the training set. The model effectively learns from data it should not have access to at prediction time. For time-series problems, use earlier data for training and later data for evaluation.
Assuming one ratio works for every dataset. A 70/15/15 split is a reasonable starting point, but it is not a universal rule. Smaller datasets may need more data in training. If you are using cross-validation, the split strategy looks different. Match the approach to your specific situation.
A high accuracy score only means something if the evaluation process was set up correctly.
Training, validation, and test sets each have a distinct role. Mixing them up, or leaking test information into development decisions, can make a model look better than it is. Getting the split right is one of the more straightforward things you can do to make your results trustworthy.