# ML Anomaly Detection Training: Start With the Base Rate > Source: > Published: 2026-09-09 14:21:31+00:00 Fitting an isolation forest takes four lines of Python. Evaluating one honestly takes labeled attack data, a defensible unit of analysis, and arithmetic that most training on ML-based anomaly detection never gets around to. That imbalance is the problem. Model fitting gets the lab time because it demos well and finishes fast. Evaluation gets a slide about precision and recall. Then the model reaches production, the queue fills with executives and backup service accounts, and the team concludes that ML does not work on their data. The scoring mechanics are already written up here: [how anomaly detection works in security ops](https://dev.to/blog/anomaly-detection-security-operations) covers the model families, and [applying anomaly detection to authentication logs](https://dev.to/blog/anomaly-detection-authentication-logs) covers per-account baselines. This is about what a curriculum has to teach around them. The first exercise should be arithmetic on paper, before anyone imports scikit-learn. Los Alamos National Laboratory released [58 days of authentication and network records](https://csr.lanl.gov/data/cyber1/) from its internal enterprise network: roughly 1.05 billion authentication events across 12,425 users, with 749 events labeled as red team activity. The base rate of malicious authentication is about seven in ten million. Run a detector over that at a false positive rate of 0.1%, which sounds like a good number in a vendor briefing. You get about 1,050,000 false positives, or 18,000 alerts a day. Catch 90% of the red team and you get 674 true positives. Precision is 0.064%: one real finding per 1,560 alerts. To reach a precision near 40% on that data, the detector needs a false positive rate around one in a million. Three orders of magnitude better than the figure on the slide. Stefan Axelsson published this argument in 2000 in [The Base-Rate Fallacy and the Difficulty of Intrusion Detection](https://dl.acm.org/doi/10.1145/357830.357849) (ACM TISSEC 3(3)). The models have changed since. The arithmetic has not, and a course that skips it produces graduates who tune toward recall and are surprised by the queue. There is a cheaper lever than model quality, and it is the row definition. Score raw events and the denominator is 1.05 billion. Aggregate the same data into user-days and it is 12,425 times 58, about 720,650 rows. At an unchanged 0.1% false positive rate that is roughly 721 false positives across the whole 58 days, around 12 a day, while the 749 red team events collapse into a much smaller number of compromised user-days. Same model, same false positive rate, a detection that a two-person team can actually work. The cost is real: you lose event-level localization, and short-lived activity that starts and finishes inside one bucket stops standing out. Choosing the bucket is a modeling decision with a precision consequence, which is why it belongs in the syllabus next to feature engineering rather than in a footnote about data prep. Whatever the course reports, it should not be accuracy, and it should not be ROC AUC alone. Both are insensitive to the base rate that dominates the result. Pick k from analyst capacity, then measure: ``` python import numpy as np from sklearn.metrics import average_precision_score def precision_at_k(scores, labels, k): # scores from score_samples(): lower is more anomalous top = np.argsort(scores)[:k] return labels[top].sum() / k for k in (10, 40, 100, 500): p = precision_at_k(scores, labels, k) print(f"k={k:4d} precision={p:.3f} found={int(p * k)}") print("average precision:", average_precision_score(labels, -scores)) ``` [`average_precision_score`](https://scikit-learn.org/stable/modules/generated/sklearn.metrics.average_precision_score.html) summarizes the precision-recall curve and moves when the base rate moves, which is the behavior you want from a headline metric here. The per-k table is what you show the SOC manager, because it answers the only question they asked: if my analysts work 40 of these a day, how many are real? Fit on an earlier window and score forward. Fitting and evaluating on the same window inflates every number in that table, for the same reason a random split inflates a malware classifier, which we covered in [ML for malware and phishing detection](https://dev.to/blog/ml-malware-phishing-detection-training). `fit_predict`, the hard half was skipped. Two situations, and both are common enough to name. If the team cannot write basic Python, an anomaly detection course is premature. Our [Threat Hunting with Data Science](https://dev.to/courses/threat-hunting-data-science) course lists basic Python as its prerequisite and points people without it to [Python for Security Analysts](https://dev.to/lp/python-for-security-analysts) first, because four days is not enough to teach both a language and a modeling discipline. If the organization cannot produce 30 days of centralized authentication or network telemetry, the modeling skills have nowhere to land. Fix retention and collection first. Anomaly detection is a technique for teams that already have the data and cannot read all of it, not a substitute for having the data. Also worth saying plainly: this training does not replace rules or threat intelligence. It covers the unlabeled remainder after those have done their work. We teach the anomaly detection block of that course with half of class time in Jupyter labs, and model tuning to reduce false positives and organization-specific model creation are two of its eight listed topics for the reason above: the precision arithmetic, not the model call, is where the detection gets built.