# Why Accuracy is Useless for Enterprise Anomaly Detection And What to Use Instead > Source: > Published: 2026-08-10 12:01:02+00:00 Here’s a test. Take a fraud detection model. Have it predict “not fraud” for every single transaction it sees. Zero exceptions. Never flag anything. On a typical FinTech dataset, that model will score **98.3% accuracy**. It is also completely useless. It will cost you millions. This is the first trap data scientists fall into when they pick up an imbalanced classification problem. Accuracy rewards the majority class. In fraud detection, the majority class is “normal.” So a model that does nothing, quite literally nothing. Looks great on the leaderboard. The metric is lying to you. In real FinTech transactional data, fraud is rare by design. Payment networks, velocity rules, and CVV checks knock out a substantial chunk of fraud before it ever reaches your ML pipeline. What’s left is a signal buried under a mountain of noise. When I was doing exploratory data analysis on the dataset for this architecture build, the skew was stark. In my payment-fraud-ml pipeline, fraud accounts for just **1.72% of all transactions, **a ratio of roughly 57:1 in favour of legitimate activity. This isn’t a quirk of a particular dataset. It’s structural. Card fraud, ACH anomalies, synthetic identity fraud and all of them are low-frequency, high-consequence events. If you train a gradient boosting model on this raw distribution without adjusting for it, the model learns to predict “normal” almost all the time. It gets rewarded for that behaviour by the accuracy metric. And it misses most of the fraud. The problem is not the algorithm. The problem is the objective function. Forget ROC-AUC for this problem. ROC-AUC measures how well the model separates classes across all thresholds, but it’s insensitive to class imbalance. When the negative class overwhelms the positive, ROC-AUC stays optimistically high even when precision on the fraud class collapses. **Precision-Recall AUC (PR-AUC)** directly measures the trade-off between: ● **Precision:** Of all the transactions I flagged as fraud, how many actually were? ● **Recall:** Of all the actual fraud cases, how many did I catch? The PR curve plots these two against each other as you shift the classification threshold. A model that genuinely learns to detect rare fraud will have a high area under that curve. A model gaming accuracy will collapse. But here’s the business translation that matters most. Every threshold decision is a business decision in disguise. Shift the threshold too high (conservative flagging), and you get low false positives but fraud slips through and hits your P&L. Shift it too low (aggressive flagging), and you block legitimate customer transactions, generating friction, chargebacks, and support costs. The right threshold is never a pure data science decision. It is set in collaboration with Risk, Compliance, and Product. The data scientist’s job is to make the trade-off visible and quantifiable. PR-AUC gives you the map. The business chooses the destination. For the model itself, I used **XGBoost** gradient-boosted decision trees that handle tabular data and sparse, skewed distributions well. The architectural choice here isn’t exotic. XGBoost works. What matters is how you train it when the problem is imbalanced. Standard hyperparameter tuning via grid search is a waste of compute on this problem space. The search space for XGBoost is wide: `max_depth`, `learning_rate`, `scale_pos_weight` (critical for class imbalance), `n_estimators`, `subsample`, `colsample_bytree`, `min_child_weight`. Grid search becomes combinatorially expensive and doesn’t adapt based on what it learns. I used **Optuna** for Bayesian hyperparameter optimization. Optuna builds a probabilistic model of the objective function and guides the search toward regions of the hyperparameter space that are more likely to improve results. Each trial informs the next. My pipeline ran 50 Optuna trials on CPU in approximately 15 minutes. **Critically: the optimization objective was set to maximize PR-AUC score. Not accuracy. Not F1. PR-AUC.** ``` python def objective(trial):params = {"max_depth": trial.suggest_int("max_depth", 3, 9),"learning_rate": trial.suggest_float("learning_rate", 1e-3, 0.3, log=True),"n_estimators": trial.suggest_int("n_estimators", 100, 1000),"scale_pos_weight": trial.suggest_float("scale_pos_weight", 1, 100),"subsample": trial.suggest_float("subsample", 0.5, 1.0),"colsample_bytree": trial.suggest_float("colsample_bytree", 0.5, 1.0),}model = XGBClassifier(**params, use_label_encoder=False,eval_metric="aucpr")model.fit(X_train, y_train)y_proba = model.predict_proba(X_val)[:, 1]return average_precision_score(y_val, y_proba) # PR-AUCstudy = optuna.create_study(direction="maximize")study.optimize(objective, n_trials=50) ``` The optimizer found `scale_pos_weight` configurations that accuracy-focused tuning would never have prioritized. That parameter alone meaningfully shifted the model’s sensitivity to the minority class. Final result: **PR-AUC of 0.923**. A high PR-AUC doesn’t get you to production in a regulated environment. **GDPR Article 22** gives data subjects the right to meaningful explanation when automated systems make decisions that significantly affect them. Blocking a credit card payment qualifies. A black-box “the model said so” answer is not a legal defence. Beyond compliance, black-box models erode trust internally. If Risk Management can’t understand why the model flagged a transaction, they won’t trust it enough to rely on it operationally. And they shouldn’t. I integrated **SHAP (SHapley Additive exPlanations)** to crack open the XGBoost model. SHAP values decompose each prediction into per-feature contributions, grounded in cooperative game theory. Every flagged transaction gets a local explanation showing exactly which features pushed the score toward fraud and by how much. The top drivers align with what experienced fraud analysts already know: ● **Transaction velocity:** Multiple transactions in rapid succession from the same card or device fingerprint ● **Night velocity interaction:** Off-hours activity combined with high transaction speed which is the highest-signal feature pair ● **Card testing micro-transactions:** High volume of sub-£10 transactions immediately preceding high-value attempts (≥ £4,500) ● **Amount z-score anomaly:** Standardized deviation from the cardholder’s historical spend baseline ● **Previous failure risk:** Unsettled account with recent failed transactions a strong precursor signal When these drivers appear in the SHAP output, they’re not just ML features. They’re audit-ready evidence. The compliance team can document them. The fraud analyst can validate them. The customer service rep can use them to explain a blocked transaction to a legitimate customer. This is how ML moves from prototype to production. The architecture covered here: class imbalance handling, PR-AUC optimization, Optuna-driven hyperparameter search, and SHAP explainability is the core of a production-grade fraud detection system, not an academic exercise. If you take one thing from this: **your optimization objective is a product decision, not a modelling preference.** In FinTech anomaly detection, accuracy is a vanity metric. PR-AUC is your north star. The threshold you pick at inference time determines whether your system protects revenue or destroys customer trust. [Why Accuracy is Useless for Enterprise Anomaly Detection And What to Use Instead](https://pub.towardsai.net/why-accuracy-is-useless-for-enterprise-anomaly-detection-and-what-to-use-instead-4512e1213f5b) was originally published in [Towards AI](https://pub.towardsai.net) on Medium, where people are continuing the conversation by highlighting and responding to this story.