# Fix Predictive Alerting False Positives in Prometheus Python Models > Source: > Published: 2026-07-10 07:02:09+00:00 *Originally published on kuryzhev.cloud* Your anomaly model tested perfectly offline — 94% precision on a historical CSV export, clean ROC curve, the works. Then it went live wired to real `query_range` calls, and within a day it was paging on-call for pod restarts and empty scrape windows. This is one of the most common failure modes we've hit rolling out predictive alerting on top of Prometheus, and it almost never shows up in the training notebook. It shows up in production, at 3am, as noise. The tell is usually a mismatch between what the model was trained to expect and what it actually receives at inference time. On our setup (Prometheus 2.48.x, Python 3.11, pandas 2.1.x, scikit-learn 1.4.0) we saw four recurring patterns: `pod_name` values), not with actual traffic load.If any of these sound familiar, don't tune the model threshold first. The threshold isn't the problem — the input pipeline is. We burned two days adjusting `predict_proba` cutoffs before realizing the features themselves were garbage. Every case above traces back to the same thing: the query that built the training set doesn't behave the same way as the query that feeds live inference. A few specific mechanisms: `step` , inference queries at another. Different aggregation windows, different NaN alignment, different feature values for the "same" metric.`_total` counters straight into the model instead of `rate()` /`increase()` . A pod restart resets the counter to zero — the model sees a massive drop that looks exactly like an anomaly, because to a naive feature vector, it is one.`predict()` or silently poison the row.`pod_name` or `request_id` can turn a 50-series query into 5,000+ series after a rolling deploy. The scorer wasn't trained for that shape and either errors out or, worse, quietly reshapes and feeds nonsense into the model.We also got bitten by the `prometheus-api-client` (pip, v0.5.3) `MetricRangeDataFrame` helper — it silently drops metrics with mismatched label sets. No exception, no warning. It just returns fewer columns than you expect, and the model happily scores whatever it gets. Make the live query mathematically equivalent to the one that trained the model. This is non-negotiable and should be treated as a contract, not a tunable. `step` to the exact resampling interval used in training (e.g. `5m` , not `15s` ). Document it as a named constant, not a magic string scattered across scripts.`rate(metric[5m])` at query time, in PromQL, before they ever reach Python.`ffill(limit=2)` , but hard-fail on longer gaps. Don't zero-fill and don't alert blind — skipping a scoring cycle is safer than scoring garbage.Gotcha: `rate()` needs at least two samples inside the range. If your scrape interval and range window are both 5m, you'll get NaN back, not an error. Always keep the range at least 2x the scrape interval. Here's the scorer with these constraints baked in: ``` bash #!/usr/bin/env python3 # scorer.py — pulls aligned Prometheus features and runs predictive alerting inference import hashlib import sys import time from datetime import datetime, timedelta import joblib import pandas as pd import requests PROM_URL = "https://prometheus.internal:9090" STEP = "5m" # MUST match training resampling interval — do not change casually LOOKBACK = timedelta(hours=2) FEATURE_QUERIES = { # use recording rules, not raw metrics, to avoid schema drift "cpu_rate": 'job:cpu_usage:rate5m{namespace="prod"}', "err_rate": 'job:http_errors:rate5m{namespace="prod"}', "req_rate": 'job:http_requests:rate5m{namespace="prod"}', } EXPECTED_FEATURE_HASH = "9a1c7f...trunc" # computed once at training time def fetch_range(promql: str, start, end): resp = requests.get( f"{PROM_URL}/api/v1/query_range", params={"query": promql, "start": start.timestamp(), "end": end.timestamp(), "step": STEP}, timeout=10, # fail fast — don't hang the whole pipeline on one bad query verify="/etc/ssl/certs/prom-ca.pem", ) resp.raise_for_status() result = resp.json()["data"]["result"] if not result: raise ValueError(f"empty result for query: {promql}") if len(result) > 1: raise ValueError(f"cardinality guard tripped: {len(result)} series for {promql}") return pd.Series( {pd.to_datetime(ts, unit="s"): float(val) if val != "NaN" else float("nan") for ts, val in result[0]["values"]} ) def build_feature_frame(): end = datetime.utcnow() start = end - LOOKBACK series = {name: fetch_range(q, start, end) for name, q in FEATURE_QUERIES.items()} df = pd.DataFrame(series) df = df.ffill(limit=2) # tolerate short scrape gaps only if df.isna().any().any(): raise RuntimeError("unrecoverable data gap — skipping inference, not alerting blind") return df def verify_feature_contract(df): computed = hashlib.sha256(",".join(sorted(df.columns)).encode()).hexdigest()[:8] if computed != EXPECTED_FEATURE_HASH[:8]: raise RuntimeError(f"feature schema drift: {computed} != {EXPECTED_FEATURE_HASH[:8]}") if __name__ == "__main__": df = build_feature_frame() verify_feature_contract(df) model = joblib.load("model_v3_feat9a1c_skl1.4.0.pkl") score = model.predict_proba(df.tail(1))[0][1] if score > 0.85: requests.post("http://alertmanager:9093/api/v2/alerts", json=[{ "labels": {"alertname": "PredictiveAnomaly", "severity": "warning"}, "annotations": {"score": str(round(score, 3))}, }], timeout=5) # keep well under Alertmanager's retry window ``` This is the fix that saved us the most future pain. Once you've aligned queries, someone will still eventually rename a metric or edit a relabel config, and the model input will silently change shape or meaning. You need to catch that before it reaches `predict()` , not after the alerts stop making sense. `level:metric:operation` naming convention (e.g. `job:http_requests:rate5m` ). Raw metric renames and relabeling then can't leak into the model input without a deliberate rule change.`hashlib.sha256` . On every inference run, recompute the hash and refuse to score if it diverges. This is cheap — a few microseconds — and it catches silent renames immediately.`model_v3_feat9a1c_skl1.4.0.pkl` . sklearn 1.4.0 loading a pickle saved under 1.3.x throws `InconsistentVersionWarning` , and if you're not watching logs closely that warning gets ignored until predictions quietly drift.Recording rules and the failure output when the hash check trips: ``` # recording_rules.yml — stabilizes feature inputs against relabeling/rename churn groups: - name: ml-feature-inputs interval: 5m # MUST equal STEP in scorer.py rules: - record: job:http_errors:rate5m expr: sum by (namespace) (rate(http_requests_total{status=~"5.."}[5m])) - record: job:http_requests:rate5m expr: sum by (namespace) (rate(http_requests_total[5m])) - record: job:cpu_usage:rate5m expr: sum by (namespace) (rate(container_cpu_usage_seconds_total[5m])) # --- Example failure output when schema drift check trips --- # $ python scorer.py # Traceback (most recent call last): # ... # RuntimeError: feature schema drift: 3fbb21a4 != 9a1c7f00 # (cause: someone renamed job:cpu_usage:rate5m -> job:cpu:rate5m in a rules PR) ``` Watch out: a joblib-saved model loaded in a container with a different libopenblas/numpy build under the hood can produce silently different float precision — same code, subtly different scores. Pin the base image by digest, not just by tag, if your scoring depends on tight probability thresholds. The third failure mode is operational, not statistical: an unbounded query can crash the scorer or hammer the Prometheus server before the model ever gets involved. `{namespace="prod", pod=~"api-.*"}` ) instead of open wildcards. Reject and log any query returning more series than the model's trained feature width — don't try to reshape and guess.`--query.timeout` (default 2m) and set a client-side timeout that's comfortably under half of Alertmanager's webhook retry window. If the scorer hangs, Alertmanager retries the webhook and you get duplicate alerts, or the notification just gets dropped silently.`query_range` call — that's not a training pipeline, that's a denial-of-service against your own monitoring stack.We also lock the Prometheus HTTP API behind mTLS or an authenticated reverse proxy — see the official [Prometheus HTTP API docs](https://prometheus.io/docs/prometheus/latest/querying/api/) for what's exposed by default. An open `/api/v1/query_range` endpoint lets anyone with pod network access read internal metric names and values, which becomes a real exfil risk the moment external services depend on it for predictive alerting. Once the pipeline is stable, the job shifts to keeping it honest over time. Three things we now run as standard practice on every predictive alerting deployment: `ml_scorer_input_freshness_seconds` and `ml_scorer_feature_hash_mismatch_total` . Alert on the alerting system — if the scorer silently stops running, you want to know before "no anomalies" gets misread as "everything's fine." Alertmanager's default `group_wait` /`repeat_interval` backoff can mask exactly that kind of outage.Predictive alerting on Prometheus data can genuinely cut noise and catch things rule-based thresholds miss — but only if the feature pipeline is treated as strictly as the model itself. Get the query semantics, schema contract, and cardinality bounds locked down first. The model tuning is the easy part; it just doesn't feel that way until the pipeline underneath it stops lying.