{"slug": "ml-anomaly-detection-training-start-with-the-base-rate", "title": "ML Anomaly Detection Training: Start With the Base Rate", "summary": "A developer's guide to training ML anomaly detection systems argues that evaluation must start with the base rate of attacks, citing Los Alamos National Laboratory data where malicious authentication events occur at about seven in ten million. The post demonstrates that at a 0.1% false positive rate, precision is only 0.064%, and recommends aggregating events into user-days to improve precision, while cautioning against relying solely on accuracy or ROC AUC.", "body_md": "Fitting an isolation forest takes four lines of Python. Evaluating one honestly takes labeled attack data, a defensible unit of analysis, and arithmetic that most training on ML-based anomaly detection never gets around to.\n\nThat imbalance is the problem. Model fitting gets the lab time because it demos well and finishes fast. Evaluation gets a slide about precision and recall. Then the model reaches production, the queue fills with executives and backup service accounts, and the team concludes that ML does not work on their data.\n\nThe scoring mechanics are already written up here: [how anomaly detection works in security ops](https://dev.to/blog/anomaly-detection-security-operations) covers the model families, and [applying anomaly detection to authentication logs](https://dev.to/blog/anomaly-detection-authentication-logs) covers per-account baselines. This is about what a curriculum has to teach around them.\n\nThe first exercise should be arithmetic on paper, before anyone imports scikit-learn.\n\nLos Alamos National Laboratory released [58 days of authentication and network records](https://csr.lanl.gov/data/cyber1/) from its internal enterprise network: roughly 1.05 billion authentication events across 12,425 users, with 749 events labeled as red team activity. The base rate of malicious authentication is about seven in ten million.\n\nRun a detector over that at a false positive rate of 0.1%, which sounds like a good number in a vendor briefing. You get about 1,050,000 false positives, or 18,000 alerts a day. Catch 90% of the red team and you get 674 true positives. Precision is 0.064%: one real finding per 1,560 alerts.\n\nTo reach a precision near 40% on that data, the detector needs a false positive rate around one in a million. Three orders of magnitude better than the figure on the slide.\n\nStefan Axelsson published this argument in 2000 in [The Base-Rate Fallacy and the Difficulty of Intrusion Detection](https://dl.acm.org/doi/10.1145/357830.357849) (ACM TISSEC 3(3)). The models have changed since. The arithmetic has not, and a course that skips it produces graduates who tune toward recall and are surprised by the queue.\n\nThere is a cheaper lever than model quality, and it is the row definition.\n\nScore raw events and the denominator is 1.05 billion. Aggregate the same data into user-days and it is 12,425 times 58, about 720,650 rows. At an unchanged 0.1% false positive rate that is roughly 721 false positives across the whole 58 days, around 12 a day, while the 749 red team events collapse into a much smaller number of compromised user-days. Same model, same false positive rate, a detection that a two-person team can actually work.\n\nThe cost is real: you lose event-level localization, and short-lived activity that starts and finishes inside one bucket stops standing out. Choosing the bucket is a modeling decision with a precision consequence, which is why it belongs in the syllabus next to feature engineering rather than in a footnote about data prep.\n\nWhatever the course reports, it should not be accuracy, and it should not be ROC AUC alone. Both are insensitive to the base rate that dominates the result.\n\nPick k from analyst capacity, then measure:\n\n``` python\nimport numpy as np\nfrom sklearn.metrics import average_precision_score\n\ndef precision_at_k(scores, labels, k):\n    # scores from score_samples(): lower is more anomalous\n    top = np.argsort(scores)[:k]\n    return labels[top].sum() / k\n\nfor k in (10, 40, 100, 500):\n    p = precision_at_k(scores, labels, k)\n    print(f\"k={k:4d}  precision={p:.3f}  found={int(p * k)}\")\n\nprint(\"average precision:\", average_precision_score(labels, -scores))\n```\n\n[`average_precision_score`](https://scikit-learn.org/stable/modules/generated/sklearn.metrics.average_precision_score.html) summarizes the precision-recall curve and moves when the base rate moves, which is the behavior you want from a headline metric here. The per-k table is what you show the SOC manager, because it answers the only question they asked: if my analysts work 40 of these a day, how many are real?\n\nFit on an earlier window and score forward. Fitting and evaluating on the same window inflates every number in that table, for the same reason a random split inflates a malware classifier, which we covered in [ML for malware and phishing detection](https://dev.to/blog/ml-malware-phishing-detection-training).\n\n`fit_predict`, the hard half was skipped.\nTwo situations, and both are common enough to name.\n\nIf the team cannot write basic Python, an anomaly detection course is premature. Our [Threat Hunting with Data Science](https://dev.to/courses/threat-hunting-data-science) course lists basic Python as its prerequisite and points people without it to [Python for Security Analysts](https://dev.to/lp/python-for-security-analysts) first, because four days is not enough to teach both a language and a modeling discipline.\n\nIf the organization cannot produce 30 days of centralized authentication or network telemetry, the modeling skills have nowhere to land. Fix retention and collection first. Anomaly detection is a technique for teams that already have the data and cannot read all of it, not a substitute for having the data.\n\nAlso worth saying plainly: this training does not replace rules or threat intelligence. It covers the unlabeled remainder after those have done their work.\n\nWe teach the anomaly detection block of that course with half of class time in Jupyter labs, and model tuning to reduce false positives and organization-specific model creation are two of its eight listed topics for the reason above: the precision arithmetic, not the model call, is where the detection gets built.", "url": "https://wpnews.pro/news/ml-anomaly-detection-training-start-with-the-base-rate", "canonical_source": "https://dev.to/cgivre/ml-anomaly-detection-training-start-with-the-base-rate-idp", "published_at": "2026-09-09 14:21:31+00:00", "updated_at": "2026-09-09 14:41:36.812401+00:00", "lang": "en", "topics": ["machine-learning", "ai-safety", "ai-research"], "entities": ["Los Alamos National Laboratory", "Stefan Axelsson", "scikit-learn"], "alternates": {"html": "https://wpnews.pro/news/ml-anomaly-detection-training-start-with-the-base-rate", "markdown": "https://wpnews.pro/news/ml-anomaly-detection-training-start-with-the-base-rate.md", "text": "https://wpnews.pro/news/ml-anomaly-detection-training-start-with-the-base-rate.txt", "jsonld": "https://wpnews.pro/news/ml-anomaly-detection-training-start-with-the-base-rate.jsonld"}}