Your CI run turns red. You open the logs, scroll through a wall of output,
and fifteen minutes later find the answer: it's that flaky test again — the
one everyone half-recognizes but nobody's fixed. You retry the job, it goes
green, you move on. Multiply that by every engineer on the team, every week,
and it adds up to real hours spent on triage that a five-second glance
shouldn't require.
The question I wanted to answer: could the pipeline tell you this run looks unusual before a human has to dig in?
A single pass/fail bit isn't enough to build on. Pipelines fail for
structurally different reasons — a flaky test, an infra hiccup, a dependency
break, resource exhaustion — and "unusual" is relative to that pipeline's
own history, not a universal threshold. A 10-minute run might be completely
normal for one workflow and a five-alarm anomaly for another that usually
finishes in 30 seconds. Fixed thresholds and simple failure-rate alerts miss
this: they treat every pipeline the same and only catch what you already
thought to watch for.
PipelineSentinel is a small, deployable layer that sits on top of existing
CI tooling and scores each run against its own pipeline's history.
Data. It pulls run metadata straight from the GitHub Actions REST API —
run duration, pass/fail outcome, retry/attempt count, triggering event, and
branch. No log-parsing required for a first pass.
Model. The baseline model is an IsolationForest (scikit-learn) over
three signals: run duration, whether the run failed, and how many attempts
it took. IsolationForest is a good first choice here because it's
unsupervised — it doesn't need a hand-labeled set of "here's what an anomaly
looks like," which you don't have on day one — and it adapts to each
pipeline's own distribution instead of a fixed global cutoff.
python
from sklearn.ensemble import IsolationForest
FEATURE_COLUMNS = ["duration_seconds", "failed", "run_attempt"]
model = IsolationForest(contamination=contamination, random_state=42)
df["is_anomaly"] = model.fit_predict(df[FEATURE_COLUMNS]) == -1