Why accuracy alone can fool you on imbalanced datasets. Somewhere, an ML model is proudly reporting 99.4% accuracy.
The dashboard is green. The stakeholders are smiling. Someone is probably preparing the report.
Then a dangerous question appears:
“How much fraud did the model actually catch?”
The answer: zero 😟
Welcome to the accuracy trap.
Watch the full video:
Consider a simulated dataset containing 20,000 card transactions, where only 0.6% are fraudulent.
Now introduce our highly sophisticated baseline:
def lazy_model(transaction):
return "not fraud"
No training. No feature engineering. No hyperparameter tuning. No GPU trying to heat the neighbourhood.
It simply predicts “not fraud” every time.
And because almost every transaction is legitimate, the model achieves:
| Metric | Lazy Model |
|---|---|
| Accuracy | 99.4% |
| Precision | 0% |
| Recall | 0% |
| F1 score | 0% |
The model is correct most of the time but useful none of the time. It catches no fraud and probably still asks for a promotion.
Accuracy asks:
“How often was the model correct overall?”
That sounds reasonable until one class heavily outnumbers the other.
For fraud detection, two other metrics are far more revealing:
Precision asks:
“Of everything flagged as fraud, how much was actually fraud?”
Low precision means your system keeps blocking genuine customers. Congratulations—you have successfully detected someone buying groceries.
Recall asks:
“Of all the fraud that really happened, how much did we catch?”
Low recall means the fraudsters leave with the money while the model celebrates its excellent accuracy.
We trained a logistic regression model using balanced class weights.
Its results looked less impressive at first:
| Metric | Lazy Model | Logistic Regression |
|---|---|---|
| Accuracy | 99.4% | 85.3% |
| Precision | 0% | 3.1% |
| Recall | 0% | 77.8% |
| F1 score | 0% | 6.0% |
The real model has lower accuracy, but it catches almost 78% of the fraud.
So which model is better?
The Lazy Model wins the dashboard beauty contest.
The logistic regression model wins the actual fraud-detection contest.
There is usually a trade-off:
The “best” threshold is therefore not only a mathematical choice. It depends on business cost.
What is worse?
The answer depends on the system.
For imbalanced classification problems, accuracy is not useless but it is often incomplete.
Always look at:
A 99.4% accurate model can still be terrible.
Metrics do not lie but they are perfectly happy to let us misunderstand them.