{"slug": "99-4-accurate-but-still-completely-useless", "title": "99.4% Accurate but still completely useless?", "summary": "An engineer demonstrated the 'accuracy trap' in machine learning using a simulated fraud-detection dataset of 20,000 card transactions with only 0.6% fraud. A baseline model that always predicts 'not fraud' achieves 99.4% accuracy but catches zero fraud, while a logistic regression model with balanced class weights has lower accuracy (85.3%) but catches 77.8% of fraud. The example highlights the importance of precision, recall, and F1 score over accuracy for imbalanced datasets.", "body_md": "Why accuracy alone can fool you on imbalanced datasets. Somewhere, an ML model is proudly reporting 99.4% accuracy.\n\nThe dashboard is green. The stakeholders are smiling. Someone is probably preparing the report.\n\nThen a dangerous question appears:\n\n“How much fraud did the model actually catch?”\n\nThe answer: **zero** 😟\n\nWelcome to the accuracy trap.\n\n**Watch the full video:**\n\nConsider a simulated dataset containing **20,000 card transactions**, where only **0.6% are fraudulent**.\n\nNow introduce our highly sophisticated baseline:\n\n``` python\ndef lazy_model(transaction):\n    return \"not fraud\"\n```\n\nNo training. No feature engineering. No hyperparameter tuning. No GPU trying to heat the neighbourhood.\n\nIt simply predicts **“not fraud”** every time.\n\nAnd because almost every transaction is legitimate, the model achieves:\n\n| Metric | Lazy Model |\n|---|---|\n| Accuracy | 99.4% |\n| Precision | 0% |\n| Recall | 0% |\n| F1 score | 0% |\n\nThe model is correct most of the time but useful none of the time. It catches no fraud and probably still asks for a promotion.\n\nAccuracy asks:\n\n“How often was the model correct overall?”\n\nThat sounds reasonable until one class heavily outnumbers the other.\n\nFor fraud detection, two other metrics are far more revealing:\n\n**Precision** asks:\n\n“Of everything flagged as fraud, how much was actually fraud?”\n\nLow precision means your system keeps blocking genuine customers. Congratulations—you have successfully detected someone buying groceries.\n\n**Recall** asks:\n\n“Of all the fraud that really happened, how much did we catch?”\n\nLow recall means the fraudsters leave with the money while the model celebrates its excellent accuracy.\n\nWe trained a logistic regression model using balanced class weights.\n\nIts results looked less impressive at first:\n\n| Metric | Lazy Model | Logistic Regression |\n|---|---|---|\n| Accuracy | 99.4% | 85.3% |\n| Precision | 0% | 3.1% |\n| Recall | 0% | 77.8% |\n| F1 score | 0% | 6.0% |\n\nThe real model has **lower accuracy**, but it catches almost **78% of the fraud**.\n\nSo which model is better?\n\nThe Lazy Model wins the dashboard beauty contest.\n\nThe logistic regression model wins the actual fraud-detection contest.\n\nThere is usually a trade-off:\n\nThe “best” threshold is therefore not only a mathematical choice. It depends on business cost.\n\nWhat is worse?\n\nThe answer depends on the system.\n\nFor imbalanced classification problems, accuracy is not useless but it is often incomplete.\n\nAlways look at:\n\nA 99.4% accurate model can still be terrible.\n\nMetrics do not lie but they are perfectly happy to let us misunderstand them.", "url": "https://wpnews.pro/news/99-4-accurate-but-still-completely-useless", "canonical_source": "https://dev.to/rajinh24/994-accurate-but-still-completely-useless-2332", "published_at": "2026-08-03 05:51:35+00:00", "updated_at": "2026-08-03 06:10:07.851325+00:00", "lang": "en", "topics": ["machine-learning", "artificial-intelligence"], "entities": [], "alternates": {"html": "https://wpnews.pro/news/99-4-accurate-but-still-completely-useless", "markdown": "https://wpnews.pro/news/99-4-accurate-but-still-completely-useless.md", "text": "https://wpnews.pro/news/99-4-accurate-but-still-completely-useless.txt", "jsonld": "https://wpnews.pro/news/99-4-accurate-but-still-completely-useless.jsonld"}}