Fix Predictive Alerting False Positives in Prometheus Python Models

A developer at kuryzhev.cloud details common failure modes when deploying predictive alerting models on Prometheus data, where offline precision of 94% drops due to mismatches between training and live inference queries. The post identifies four recurring patterns—time alignment drift, counter reset poisoning, NaN propagation, and label explosion—and provides a hardened scorer script that enforces query equivalence, rate conversion, and explicit NaN handling to reduce false positives.

Originally published on kuryzhev.cloud Your anomaly model tested perfectly offline — 94% precision on a historical CSV export, clean ROC curve, the works. Then it went live wired to real query range calls, and within a day it was paging on-call for pod restarts and empty scrape windows. This is one of the most common failure modes we've hit rolling out predictive alerting on top of Prometheus, and it almost never shows up in the training notebook. It shows up in production, at 3am, as noise. The tell is usually a mismatch between what the model was trained to expect and what it actually receives at inference time. On our setup Prometheus 2.48.x, Python 3.11, pandas 2.1.x, scikit-learn 1.4.0 we saw four recurring patterns: pod name values , not with actual traffic load.If any of these sound familiar, don't tune the model threshold first. The threshold isn't the problem — the input pipeline is. We burned two days adjusting predict proba cutoffs before realizing the features themselves were garbage. Every case above traces back to the same thing: the query that built the training set doesn't behave the same way as the query that feeds live inference. A few specific mechanisms: step , inference queries at another. Different aggregation windows, different NaN alignment, different feature values for the "same" metric. total counters straight into the model instead of rate / increase . A pod restart resets the counter to zero — the model sees a massive drop that looks exactly like an anomaly, because to a naive feature vector, it is one. predict or silently poison the row. pod name or request id can turn a 50-series query into 5,000+ series after a rolling deploy. The scorer wasn't trained for that shape and either errors out or, worse, quietly reshapes and feeds nonsense into the model.We also got bitten by the prometheus-api-client pip, v0.5.3 MetricRangeDataFrame helper — it silently drops metrics with mismatched label sets. No exception, no warning. It just returns fewer columns than you expect, and the model happily scores whatever it gets. Make the live query mathematically equivalent to the one that trained the model. This is non-negotiable and should be treated as a contract, not a tunable. step to the exact resampling interval used in training e.g. 5m , not 15s . Document it as a named constant, not a magic string scattered across scripts. rate metric 5m at query time, in PromQL, before they ever reach Python. ffill limit=2 , but hard-fail on longer gaps. Don't zero-fill and don't alert blind — skipping a scoring cycle is safer than scoring garbage.Gotcha: rate needs at least two samples inside the range. If your scrape interval and range window are both 5m, you'll get NaN back, not an error. Always keep the range at least 2x the scrape interval. Here's the scorer with these constraints baked in: bash /usr/bin/env python3 scorer.py — pulls aligned Prometheus features and runs predictive alerting inference import hashlib import sys import time from datetime import datetime, timedelta import joblib import pandas as pd import requests PROM URL = "https://prometheus.internal:9090" STEP = "5m" MUST match training resampling interval — do not change casually LOOKBACK = timedelta hours=2 FEATURE QUERIES = { use recording rules, not raw metrics, to avoid schema drift "cpu rate": 'job:cpu usage:rate5m{namespace="prod"}', "err rate": 'job:http errors:rate5m{namespace="prod"}', "req rate": 'job:http requests:rate5m{namespace="prod"}', } EXPECTED FEATURE HASH = "9a1c7f...trunc" computed once at training time def fetch range promql: str, start, end : resp = requests.get f"{PROM URL}/api/v1/query range", params={"query": promql, "start": start.timestamp , "end": end.timestamp , "step": STEP}, timeout=10, fail fast — don't hang the whole pipeline on one bad query verify="/etc/ssl/certs/prom-ca.pem", resp.raise for status result = resp.json "data" "result" if not result: raise ValueError f"empty result for query: {promql}" if len result 1: raise ValueError f"cardinality guard tripped: {len result } series for {promql}" return pd.Series {pd.to datetime ts, unit="s" : float val if val = "NaN" else float "nan" for ts, val in result 0 "values" } def build feature frame : end = datetime.utcnow start = end - LOOKBACK series = {name: fetch range q, start, end for name, q in FEATURE QUERIES.items } df = pd.DataFrame series df = df.ffill limit=2 tolerate short scrape gaps only if df.isna .any .any : raise RuntimeError "unrecoverable data gap — skipping inference, not alerting blind" return df def verify feature contract df : computed = hashlib.sha256 ",".join sorted df.columns .encode .hexdigest :8 if computed = EXPECTED FEATURE HASH :8 : raise RuntimeError f"feature schema drift: {computed} = {EXPECTED FEATURE HASH :8 }" if name == " main ": df = build feature frame verify feature contract df model = joblib.load "model v3 feat9a1c skl1.4.0.pkl" score = model.predict proba df.tail 1 0 1 if score 0.85: requests.post "http://alertmanager:9093/api/v2/alerts", json= { "labels": {"alertname": "PredictiveAnomaly", "severity": "warning"}, "annotations": {"score": str round score, 3 }, } , timeout=5 keep well under Alertmanager's retry window This is the fix that saved us the most future pain. Once you've aligned queries, someone will still eventually rename a metric or edit a relabel config, and the model input will silently change shape or meaning. You need to catch that before it reaches predict , not after the alerts stop making sense. level:metric:operation naming convention e.g. job:http requests:rate5m . Raw metric renames and relabeling then can't leak into the model input without a deliberate rule change. hashlib.sha256 . On every inference run, recompute the hash and refuse to score if it diverges. This is cheap — a few microseconds — and it catches silent renames immediately. model v3 feat9a1c skl1.4.0.pkl . sklearn 1.4.0 loading a pickle saved under 1.3.x throws InconsistentVersionWarning , and if you're not watching logs closely that warning gets ignored until predictions quietly drift.Recording rules and the failure output when the hash check trips: recording rules.yml — stabilizes feature inputs against relabeling/rename churn groups: - name: ml-feature-inputs interval: 5m MUST equal STEP in scorer.py rules: - record: job:http errors:rate5m expr: sum by namespace rate http requests total{status=~"5.."} 5m - record: job:http requests:rate5m expr: sum by namespace rate http requests total 5m - record: job:cpu usage:rate5m expr: sum by namespace rate container cpu usage seconds total 5m --- Example failure output when schema drift check trips --- $ python scorer.py Traceback most recent call last : ... RuntimeError: feature schema drift: 3fbb21a4 = 9a1c7f00 cause: someone renamed job:cpu usage:rate5m - job:cpu:rate5m in a rules PR Watch out: a joblib-saved model loaded in a container with a different libopenblas/numpy build under the hood can produce silently different float precision — same code, subtly different scores. Pin the base image by digest, not just by tag, if your scoring depends on tight probability thresholds. The third failure mode is operational, not statistical: an unbounded query can crash the scorer or hammer the Prometheus server before the model ever gets involved. {namespace="prod", pod=~"api-. "} instead of open wildcards. Reject and log any query returning more series than the model's trained feature width — don't try to reshape and guess. --query.timeout default 2m and set a client-side timeout that's comfortably under half of Alertmanager's webhook retry window. If the scorer hangs, Alertmanager retries the webhook and you get duplicate alerts, or the notification just gets dropped silently. query range call — that's not a training pipeline, that's a denial-of-service against your own monitoring stack.We also lock the Prometheus HTTP API behind mTLS or an authenticated reverse proxy — see the official Prometheus HTTP API docs https://prometheus.io/docs/prometheus/latest/querying/api/ for what's exposed by default. An open /api/v1/query range endpoint lets anyone with pod network access read internal metric names and values, which becomes a real exfil risk the moment external services depend on it for predictive alerting. Once the pipeline is stable, the job shifts to keeping it honest over time. Three things we now run as standard practice on every predictive alerting deployment: ml scorer input freshness seconds and ml scorer feature hash mismatch total . Alert on the alerting system — if the scorer silently stops running, you want to know before "no anomalies" gets misread as "everything's fine." Alertmanager's default group wait / repeat interval backoff can mask exactly that kind of outage.Predictive alerting on Prometheus data can genuinely cut noise and catch things rule-based thresholds miss — but only if the feature pipeline is treated as strictly as the model itself. Get the query semantics, schema contract, and cardinality bounds locked down first. The model tuning is the easy part; it just doesn't feel that way until the pipeline underneath it stops lying.