Originally published on kuryzhev.cloud
Your anomaly model tested perfectly offline — 94% precision on a historical CSV export, clean ROC curve, the works. Then it went live wired to real query_range
calls, and within a day it was paging on-call for pod restarts and empty scrape windows. This is one of the most common failure modes we've hit rolling out predictive alerting on top of Prometheus, and it almost never shows up in the training notebook. It shows up in production, at 3am, as noise.
The tell is usually a mismatch between what the model was trained to expect and what it actually receives at inference time. On our setup (Prometheus 2.48.x, Python 3.11, pandas 2.1.x, scikit-learn 1.4.0) we saw four recurring patterns:
pod_name
values), not with actual traffic load.If any of these sound familiar, don't tune the model threshold first. The threshold isn't the problem — the input pipeline is. We burned two days adjusting predict_proba
cutoffs before realizing the features themselves were garbage.
Every case above traces back to the same thing: the query that built the training set doesn't behave the same way as the query that feeds live inference. A few specific mechanisms:
step
, inference queries at another. Different aggregation windows, different NaN alignment, different feature values for the "same" metric._total
counters straight into the model instead of rate()
/increase()
. A pod restart resets the counter to zero — the model sees a massive drop that looks exactly like an anomaly, because to a naive feature vector, it is one.predict()
or silently poison the row.pod_name
or request_id
can turn a 50-series query into 5,000+ series after a rolling deploy. The scorer wasn't trained for that shape and either errors out or, worse, quietly reshapes and feeds nonsense into the model.We also got bitten by the prometheus-api-client
(pip, v0.5.3) MetricRangeDataFrame
helper — it silently drops metrics with mismatched label sets. No exception, no warning. It just returns fewer columns than you expect, and the model happily scores whatever it gets.
Make the live query mathematically equivalent to the one that trained the model. This is non-negotiable and should be treated as a contract, not a tunable.
step
to the exact resampling interval used in training (e.g. 5m
, not 15s
). Document it as a named constant, not a magic string scattered across scripts.rate(metric[5m])
at query time, in PromQL, before they ever reach Python.ffill(limit=2)
, but hard-fail on longer gaps. Don't zero-fill and don't alert blind — skipping a scoring cycle is safer than scoring garbage.Gotcha: rate()
needs at least two samples inside the range. If your scrape interval and range window are both 5m, you'll get NaN back, not an error. Always keep the range at least 2x the scrape interval.
Here's the scorer with these constraints baked in:
#!/usr/bin/env python3
import hashlib
import sys
import time
from datetime import datetime, timedelta
import joblib
import pandas as pd
import requests
PROM_URL = "https://prometheus.internal:9090"
STEP = "5m" # MUST match training resampling interval — do not change casually
LOOKBACK = timedelta(hours=2)
FEATURE_QUERIES = {
"cpu_rate": 'job:cpu_usage:rate5m{namespace="prod"}',
"err_rate": 'job:http_errors:rate5m{namespace="prod"}',
"req_rate": 'job:http_requests:rate5m{namespace="prod"}',
}
EXPECTED_FEATURE_HASH = "9a1c7f...trunc" # computed once at training time
def fetch_range(promql: str, start, end):
resp = requests.get(
f"{PROM_URL}/api/v1/query_range",
params={"query": promql, "start": start.timestamp(), "end": end.timestamp(), "step": STEP},
timeout=10, # fail fast — don't hang the whole pipeline on one bad query
verify="/etc/ssl/certs/prom-ca.pem",
)
resp.raise_for_status()
result = resp.json()["data"]["result"]
if not result:
raise ValueError(f"empty result for query: {promql}")
if len(result) > 1:
raise ValueError(f"cardinality guard tripped: {len(result)} series for {promql}")
return pd.Series(
{pd.to_datetime(ts, unit="s"): float(val) if val != "NaN" else float("nan")
for ts, val in result[0]["values"]}
)
def build_feature_frame():
end = datetime.utcnow()
start = end - LOOKBACK
series = {name: fetch_range(q, start, end) for name, q in FEATURE_QUERIES.items()}
df = pd.DataFrame(series)
df = df.ffill(limit=2) # tolerate short scrape gaps only
if df.isna().any().any():
raise RuntimeError("unrecoverable data gap — skipping inference, not alerting blind")
return df
def verify_feature_contract(df):
computed = hashlib.sha256(",".join(sorted(df.columns)).encode()).hexdigest()[:8]
if computed != EXPECTED_FEATURE_HASH[:8]:
raise RuntimeError(f"feature schema drift: {computed} != {EXPECTED_FEATURE_HASH[:8]}")
if __name__ == "__main__":
df = build_feature_frame()
verify_feature_contract(df)
model = joblib.load("model_v3_feat9a1c_skl1.4.0.pkl")
score = model.predict_proba(df.tail(1))[0][1]
if score > 0.85:
requests.post("http://alertmanager:9093/api/v2/alerts", json=[{
"labels": {"alertname": "PredictiveAnomaly", "severity": "warning"},
"annotations": {"score": str(round(score, 3))},
}], timeout=5) # keep well under Alertmanager's retry window
This is the fix that saved us the most future pain. Once you've aligned queries, someone will still eventually rename a metric or edit a relabel config, and the model input will silently change shape or meaning. You need to catch that before it reaches predict()
, not after the alerts stop making sense.
level:metric:operation
naming convention (e.g. job:http_requests:rate5m
). Raw metric renames and relabeling then can't leak into the model input without a deliberate rule change.hashlib.sha256
. On every inference run, recompute the hash and refuse to score if it diverges. This is cheap — a few microseconds — and it catches silent renames immediately.model_v3_feat9a1c_skl1.4.0.pkl
. sklearn 1.4.0 a pickle saved under 1.3.x throws InconsistentVersionWarning
, and if you're not watching logs closely that warning gets ignored until predictions quietly drift.Recording rules and the failure output when the hash check trips:
groups:
- name: ml-feature-inputs
interval: 5m # MUST equal STEP in scorer.py
rules:
- record: job:http_errors:rate5m
expr: sum by (namespace) (rate(http_requests_total{status=~"5.."}[5m]))
- record: job:http_requests:rate5m
expr: sum by (namespace) (rate(http_requests_total[5m]))
- record: job:cpu_usage:rate5m
expr: sum by (namespace) (rate(container_cpu_usage_seconds_total[5m]))
Watch out: a joblib-saved model loaded in a container with a different libopenblas/numpy build under the hood can produce silently different float precision — same code, subtly different scores. Pin the base image by digest, not just by tag, if your scoring depends on tight probability thresholds.
The third failure mode is operational, not statistical: an unbounded query can crash the scorer or hammer the Prometheus server before the model ever gets involved.
{namespace="prod", pod=~"api-.*"}
) instead of open wildcards. Reject and log any query returning more series than the model's trained feature width — don't try to reshape and guess.--query.timeout
(default 2m) and set a client-side timeout that's comfortably under half of Alertmanager's webhook retry window. If the scorer hangs, Alertmanager retries the webhook and you get duplicate alerts, or the notification just gets dropped silently.query_range
call — that's not a training pipeline, that's a denial-of-service against your own monitoring stack.We also lock the Prometheus HTTP API behind mTLS or an authenticated reverse proxy — see the official Prometheus HTTP API docs for what's exposed by default. An open /api/v1/query_range
endpoint lets anyone with pod network access read internal metric names and values, which becomes a real exfil risk the moment external services depend on it for predictive alerting.
Once the pipeline is stable, the job shifts to keeping it honest over time. Three things we now run as standard practice on every predictive alerting deployment:
ml_scorer_input_freshness_seconds
and ml_scorer_feature_hash_mismatch_total
. Alert on the alerting system — if the scorer silently stops running, you want to know before "no anomalies" gets misread as "everything's fine." Alertmanager's default group_wait
/repeat_interval
backoff can mask exactly that kind of outage.Predictive alerting on Prometheus data can genuinely cut noise and catch things rule-based thresholds miss — but only if the feature pipeline is treated as strictly as the model itself. Get the query semantics, schema contract, and cardinality bounds locked down first. The model tuning is the easy part; it just doesn't feel that way until the pipeline underneath it stops lying.