{"slug": "give-your-ci-pipeline-an-error-budget-before-adding-ai-triage", "title": "Give Your CI Pipeline an Error Budget Before Adding AI Triage", "summary": "A developer argues that AI triage for CI pipelines is ineffective without first establishing error budgets and structured evidence. The post outlines service-level indicators for queue time, execution time, infrastructure failures, and retry rates, and recommends classifying failures into categories like code, test flake, and infrastructure before introducing AI explanations.", "body_md": "An AI-generated explanation cannot rescue an unhealthy CI pipeline. If jobs wait unpredictably, runners disappear, retries hide flakes, or logs lack revision metadata, the model is summarizing unreliable evidence.\n\nStart with four service-level indicators:\n\n```\nqueue_seconds = job_started_at - job_queued_at\nexecution_seconds = job_finished_at - job_started_at\ninfra_failure_rate = infrastructure_failures / completed_jobs\nretry_recovery_rate = passed_after_retry / retried_jobs\n```\n\nSplit failures into `code`\n\n, `test_flake`\n\n, `infrastructure`\n\n, `dependency`\n\n, `policy`\n\n, and `unknown`\n\n. Do not let `unknown`\n\nbecome permanent.\n\nFor a protected-branch workflow, one starting target could be:\n\nThese are example thresholds, not universal standards. Derive yours from developer wait tolerance, release risk, runner capacity, and current distributions.\n\nIf a week has 1,000 protected-branch runs and the infrastructure-failure objective allows 10, the eleventh failure should change priorities. Pause runner-image changes, investigate the dominant signature, or reduce concurrency until recovery.\n\nInstrument queue and execution separately. Scaling executors cannot fix serialized dependency downloads; caching cannot fix insufficient runner capacity. Record retries as new attempts linked to the original job, never overwrite the initial failure.\n\nBefore AI triage, require structured evidence:\n\n```\n{\n  \"revision\": \"abc123\",\n  \"workflow_revision\": \"def456\",\n  \"runner_image\": \"ubuntu-24.04@sha256:...\",\n  \"attempt\": 2,\n  \"previous_attempt\": 1,\n  \"exit_code\": 1,\n  \"failure_class\": \"test_flake\"\n}\n```\n\nThen an AI summary can cite stable logs and metadata. Evaluate it on classification accuracy, missing-evidence rate, analyst correction time, and dangerous false reassurance—not prose quality.\n\nThe public [MonkeyCode repository](https://github.com/chaitin/MonkeyCode) describes AI tasks, development environments, and automated PR/MR review. CI evidence can feed that category of workflow, but this article reports no MonkeyCode deployment measurement or integration test.\n\nDisclosure: I contribute to the MonkeyCode project. The public repository provides context; the SLO framework is independent.\n\nOperate the evidence first. Automate the explanation second.", "url": "https://wpnews.pro/news/give-your-ci-pipeline-an-error-budget-before-adding-ai-triage", "canonical_source": "https://dev.to/odd_background_328/give-your-ci-pipeline-an-error-budget-before-adding-ai-triage-3j4g", "published_at": "2026-07-12 15:12:23+00:00", "updated_at": "2026-07-12 15:15:07.733508+00:00", "lang": "en", "topics": ["developer-tools", "mlops", "ai-agents"], "entities": ["MonkeyCode"], "alternates": {"html": "https://wpnews.pro/news/give-your-ci-pipeline-an-error-budget-before-adding-ai-triage", "markdown": "https://wpnews.pro/news/give-your-ci-pipeline-an-error-budget-before-adding-ai-triage.md", "text": "https://wpnews.pro/news/give-your-ci-pipeline-an-error-budget-before-adding-ai-triage.txt", "jsonld": "https://wpnews.pro/news/give-your-ci-pipeline-an-error-budget-before-adding-ai-triage.jsonld"}}