{"slug": "fix-predictive-alerting-false-positives-in-prometheus-python-models", "title": "Fix Predictive Alerting False Positives in Prometheus Python Models", "summary": "A developer at kuryzhev.cloud details common failure modes when deploying predictive alerting models on Prometheus data, where offline precision of 94% drops due to mismatches between training and live inference queries. The post identifies four recurring patterns—time alignment drift, counter reset poisoning, NaN propagation, and label explosion—and provides a hardened scorer script that enforces query equivalence, rate conversion, and explicit NaN handling to reduce false positives.", "body_md": "*Originally published on kuryzhev.cloud*\n\nYour anomaly model tested perfectly offline — 94% precision on a historical CSV export, clean ROC curve, the works. Then it went live wired to real `query_range`\n\ncalls, and within a day it was paging on-call for pod restarts and empty scrape windows. This is one of the most common failure modes we've hit rolling out predictive alerting on top of Prometheus, and it almost never shows up in the training notebook. It shows up in production, at 3am, as noise.\n\nThe tell is usually a mismatch between what the model was trained to expect and what it actually receives at inference time. On our setup (Prometheus 2.48.x, Python 3.11, pandas 2.1.x, scikit-learn 1.4.0) we saw four recurring patterns:\n\n`pod_name`\n\nvalues), not with actual traffic load.If any of these sound familiar, don't tune the model threshold first. The threshold isn't the problem — the input pipeline is. We burned two days adjusting `predict_proba`\n\ncutoffs before realizing the features themselves were garbage.\n\nEvery case above traces back to the same thing: the query that built the training set doesn't behave the same way as the query that feeds live inference. A few specific mechanisms:\n\n`step`\n\n, inference queries at another. Different aggregation windows, different NaN alignment, different feature values for the \"same\" metric.`_total`\n\ncounters straight into the model instead of `rate()`\n\n/`increase()`\n\n. A pod restart resets the counter to zero — the model sees a massive drop that looks exactly like an anomaly, because to a naive feature vector, it is one.`predict()`\n\nor silently poison the row.`pod_name`\n\nor `request_id`\n\ncan turn a 50-series query into 5,000+ series after a rolling deploy. The scorer wasn't trained for that shape and either errors out or, worse, quietly reshapes and feeds nonsense into the model.We also got bitten by the `prometheus-api-client`\n\n(pip, v0.5.3) `MetricRangeDataFrame`\n\nhelper — it silently drops metrics with mismatched label sets. No exception, no warning. It just returns fewer columns than you expect, and the model happily scores whatever it gets.\n\nMake the live query mathematically equivalent to the one that trained the model. This is non-negotiable and should be treated as a contract, not a tunable.\n\n`step`\n\nto the exact resampling interval used in training (e.g. `5m`\n\n, not `15s`\n\n). Document it as a named constant, not a magic string scattered across scripts.`rate(metric[5m])`\n\nat query time, in PromQL, before they ever reach Python.`ffill(limit=2)`\n\n, but hard-fail on longer gaps. Don't zero-fill and don't alert blind — skipping a scoring cycle is safer than scoring garbage.Gotcha: `rate()`\n\nneeds at least two samples inside the range. If your scrape interval and range window are both 5m, you'll get NaN back, not an error. Always keep the range at least 2x the scrape interval.\n\nHere's the scorer with these constraints baked in:\n\n``` bash\n#!/usr/bin/env python3\n# scorer.py — pulls aligned Prometheus features and runs predictive alerting inference\nimport hashlib\nimport sys\nimport time\nfrom datetime import datetime, timedelta\n\nimport joblib\nimport pandas as pd\nimport requests\n\nPROM_URL = \"https://prometheus.internal:9090\"\nSTEP = \"5m\"              # MUST match training resampling interval — do not change casually\nLOOKBACK = timedelta(hours=2)\nFEATURE_QUERIES = {\n    # use recording rules, not raw metrics, to avoid schema drift\n    \"cpu_rate\": 'job:cpu_usage:rate5m{namespace=\"prod\"}',\n    \"err_rate\": 'job:http_errors:rate5m{namespace=\"prod\"}',\n    \"req_rate\": 'job:http_requests:rate5m{namespace=\"prod\"}',\n}\nEXPECTED_FEATURE_HASH = \"9a1c7f...trunc\"  # computed once at training time\n\ndef fetch_range(promql: str, start, end):\n    resp = requests.get(\n        f\"{PROM_URL}/api/v1/query_range\",\n        params={\"query\": promql, \"start\": start.timestamp(), \"end\": end.timestamp(), \"step\": STEP},\n        timeout=10,  # fail fast — don't hang the whole pipeline on one bad query\n        verify=\"/etc/ssl/certs/prom-ca.pem\",\n    )\n    resp.raise_for_status()\n    result = resp.json()[\"data\"][\"result\"]\n    if not result:\n        raise ValueError(f\"empty result for query: {promql}\")\n    if len(result) > 1:\n        raise ValueError(f\"cardinality guard tripped: {len(result)} series for {promql}\")\n    return pd.Series(\n        {pd.to_datetime(ts, unit=\"s\"): float(val) if val != \"NaN\" else float(\"nan\")\n         for ts, val in result[0][\"values\"]}\n    )\n\ndef build_feature_frame():\n    end = datetime.utcnow()\n    start = end - LOOKBACK\n    series = {name: fetch_range(q, start, end) for name, q in FEATURE_QUERIES.items()}\n    df = pd.DataFrame(series)\n    df = df.ffill(limit=2)          # tolerate short scrape gaps only\n    if df.isna().any().any():\n        raise RuntimeError(\"unrecoverable data gap — skipping inference, not alerting blind\")\n    return df\n\ndef verify_feature_contract(df):\n    computed = hashlib.sha256(\",\".join(sorted(df.columns)).encode()).hexdigest()[:8]\n    if computed != EXPECTED_FEATURE_HASH[:8]:\n        raise RuntimeError(f\"feature schema drift: {computed} != {EXPECTED_FEATURE_HASH[:8]}\")\n\nif __name__ == \"__main__\":\n    df = build_feature_frame()\n    verify_feature_contract(df)\n    model = joblib.load(\"model_v3_feat9a1c_skl1.4.0.pkl\")\n    score = model.predict_proba(df.tail(1))[0][1]\n    if score > 0.85:\n        requests.post(\"http://alertmanager:9093/api/v2/alerts\", json=[{\n            \"labels\": {\"alertname\": \"PredictiveAnomaly\", \"severity\": \"warning\"},\n            \"annotations\": {\"score\": str(round(score, 3))},\n        }], timeout=5)  # keep well under Alertmanager's retry window\n```\n\nThis is the fix that saved us the most future pain. Once you've aligned queries, someone will still eventually rename a metric or edit a relabel config, and the model input will silently change shape or meaning. You need to catch that before it reaches `predict()`\n\n, not after the alerts stop making sense.\n\n`level:metric:operation`\n\nnaming convention (e.g. `job:http_requests:rate5m`\n\n). Raw metric renames and relabeling then can't leak into the model input without a deliberate rule change.`hashlib.sha256`\n\n. On every inference run, recompute the hash and refuse to score if it diverges. This is cheap — a few microseconds — and it catches silent renames immediately.`model_v3_feat9a1c_skl1.4.0.pkl`\n\n. sklearn 1.4.0 loading a pickle saved under 1.3.x throws `InconsistentVersionWarning`\n\n, and if you're not watching logs closely that warning gets ignored until predictions quietly drift.Recording rules and the failure output when the hash check trips:\n\n```\n# recording_rules.yml — stabilizes feature inputs against relabeling/rename churn\ngroups:\n  - name: ml-feature-inputs\n    interval: 5m           # MUST equal STEP in scorer.py\n    rules:\n      - record: job:http_errors:rate5m\n        expr: sum by (namespace) (rate(http_requests_total{status=~\"5..\"}[5m]))\n      - record: job:http_requests:rate5m\n        expr: sum by (namespace) (rate(http_requests_total[5m]))\n      - record: job:cpu_usage:rate5m\n        expr: sum by (namespace) (rate(container_cpu_usage_seconds_total[5m]))\n\n# --- Example failure output when schema drift check trips ---\n# $ python scorer.py\n# Traceback (most recent call last):\n#   ...\n# RuntimeError: feature schema drift: 3fbb21a4 != 9a1c7f00\n# (cause: someone renamed job:cpu_usage:rate5m -> job:cpu:rate5m in a rules PR)\n```\n\nWatch out: a joblib-saved model loaded in a container with a different libopenblas/numpy build under the hood can produce silently different float precision — same code, subtly different scores. Pin the base image by digest, not just by tag, if your scoring depends on tight probability thresholds.\n\nThe third failure mode is operational, not statistical: an unbounded query can crash the scorer or hammer the Prometheus server before the model ever gets involved.\n\n`{namespace=\"prod\", pod=~\"api-.*\"}`\n\n) instead of open wildcards. Reject and log any query returning more series than the model's trained feature width — don't try to reshape and guess.`--query.timeout`\n\n(default 2m) and set a client-side timeout that's comfortably under half of Alertmanager's webhook retry window. If the scorer hangs, Alertmanager retries the webhook and you get duplicate alerts, or the notification just gets dropped silently.`query_range`\n\ncall — that's not a training pipeline, that's a denial-of-service against your own monitoring stack.We also lock the Prometheus HTTP API behind mTLS or an authenticated reverse proxy — see the official [Prometheus HTTP API docs](https://prometheus.io/docs/prometheus/latest/querying/api/) for what's exposed by default. An open `/api/v1/query_range`\n\nendpoint lets anyone with pod network access read internal metric names and values, which becomes a real exfil risk the moment external services depend on it for predictive alerting.\n\nOnce the pipeline is stable, the job shifts to keeping it honest over time. Three things we now run as standard practice on every predictive alerting deployment:\n\n`ml_scorer_input_freshness_seconds`\n\nand `ml_scorer_feature_hash_mismatch_total`\n\n. Alert on the alerting system — if the scorer silently stops running, you want to know before \"no anomalies\" gets misread as \"everything's fine.\" Alertmanager's default `group_wait`\n\n/`repeat_interval`\n\nbackoff can mask exactly that kind of outage.Predictive alerting on Prometheus data can genuinely cut noise and catch things rule-based thresholds miss — but only if the feature pipeline is treated as strictly as the model itself. Get the query semantics, schema contract, and cardinality bounds locked down first. The model tuning is the easy part; it just doesn't feel that way until the pipeline underneath it stops lying.", "url": "https://wpnews.pro/news/fix-predictive-alerting-false-positives-in-prometheus-python-models", "canonical_source": "https://dev.to/oleksandr_kuryzhev_42873f/fix-predictive-alerting-false-positives-in-prometheus-python-models-45dh", "published_at": "2026-07-10 07:02:09+00:00", "updated_at": "2026-07-10 07:13:10.321505+00:00", "lang": "en", "topics": ["machine-learning", "mlops", "developer-tools", "artificial-intelligence"], "entities": ["Prometheus", "Python", "scikit-learn", "pandas", "kuryzhev.cloud", "prometheus-api-client"], "alternates": {"html": "https://wpnews.pro/news/fix-predictive-alerting-false-positives-in-prometheus-python-models", "markdown": "https://wpnews.pro/news/fix-predictive-alerting-false-positives-in-prometheus-python-models.md", "text": "https://wpnews.pro/news/fix-predictive-alerting-false-positives-in-prometheus-python-models.txt", "jsonld": "https://wpnews.pro/news/fix-predictive-alerting-false-positives-in-prometheus-python-models.jsonld"}}