Give Your CI Pipeline an Error Budget Before Adding AI Triage

A developer argues that AI triage for CI pipelines is ineffective without first establishing error budgets and structured evidence. The post outlines service-level indicators for queue time, execution time, infrastructure failures, and retry rates, and recommends classifying failures into categories like code, test flake, and infrastructure before introducing AI explanations.

An AI-generated explanation cannot rescue an unhealthy CI pipeline. If jobs wait unpredictably, runners disappear, retries hide flakes, or logs lack revision metadata, the model is summarizing unreliable evidence. Start with four service-level indicators: queue seconds = job started at - job queued at execution seconds = job finished at - job started at infra failure rate = infrastructure failures / completed jobs retry recovery rate = passed after retry / retried jobs Split failures into code , test flake , infrastructure , dependency , policy , and unknown . Do not let unknown become permanent. For a protected-branch workflow, one starting target could be: These are example thresholds, not universal standards. Derive yours from developer wait tolerance, release risk, runner capacity, and current distributions. If a week has 1,000 protected-branch runs and the infrastructure-failure objective allows 10, the eleventh failure should change priorities. Pause runner-image changes, investigate the dominant signature, or reduce concurrency until recovery. Instrument queue and execution separately. Scaling executors cannot fix serialized dependency downloads; caching cannot fix insufficient runner capacity. Record retries as new attempts linked to the original job, never overwrite the initial failure. Before AI triage, require structured evidence: { "revision": "abc123", "workflow revision": "def456", "runner image": "ubuntu-24.04@sha256:...", "attempt": 2, "previous attempt": 1, "exit code": 1, "failure class": "test flake" } Then an AI summary can cite stable logs and metadata. Evaluate it on classification accuracy, missing-evidence rate, analyst correction time, and dangerous false reassurance—not prose quality. The public MonkeyCode repository https://github.com/chaitin/MonkeyCode describes AI tasks, development environments, and automated PR/MR review. CI evidence can feed that category of workflow, but this article reports no MonkeyCode deployment measurement or integration test. Disclosure: I contribute to the MonkeyCode project. The public repository provides context; the SLO framework is independent. Operate the evidence first. Automate the explanation second.