Your CI Pipeline Was Always Production

Jerry Orr's essay argues that development pipelines, including CI/CD and QA environments, should be treated as production systems with outage-level urgency, a view echoed by Hacker News commenters. The argument is supported by Google's 2016 data showing 1.5% of test runs are flaky and 16% of tests exhibit flakiness, and by Buildkite's advocacy for applying SRE principles to CI/CD. The urgency is heightened by the January 2023 CircleCI breach and the rise of coding agents that depend on pipeline availability.

Dev Tools https://sourcefeed.dev/c/dev-tools Article Your CI Pipeline Was Always Production SLOs, error budgets, and owners for dev infrastructure — and why coding agents just made the case urgent. Lenn Voss https://sourcefeed.dev/u/lennart voss Jerry Orr's short essay "The development pipeline is a production system" https://sundry.jerryorr.com/2026/07/31/development-pipeline-is-a-production-system makes an argument most engineers will nod along to: when prod goes down, it's all-hands-on-deck, but when the build breaks, CI queues back up for an hour, or the QA environment falls over, teams shrug and wait. His point is that both events stop the company from shipping value, so both deserve outage-level urgency. He's right, and the Hacker News crowd mostly agreed — with the predictable objection that calling everything "production" makes the word useless. That objection deserves a real answer, because the interesting question isn't whether your pipeline is production. It's which production practices actually transfer, and which ones turn into 2 a.m. pages because a linter was slow. This idea has a track record Orr's framing feels fresh, but the industry has been converging on it for a decade under different names. Google's internal test infrastructure has been run like a production service since at least the mid-2010s — its testing team published hard numbers back in 2016 showing about 1.5% of all test runs report a flaky result and nearly 16% of tests exhibit some flakiness https://testing.googleblog.com/2016/05/flaky-tests-at-google-and-how-we.html , and built dedicated systems to quarantine and track them. You don't collect fleet-wide flakiness telemetry unless you already believe the test infrastructure is a service with users. The vendors got there too. Buildkite has argued for applying SRE discipline directly to CI/CD https://buildkite.com/resources/blog/applying-sre-principles-to-cicd/ — SLOs, SLIs, and error budgets for pipelines, with concrete targets like "builds start within one minute" and "developers get test feedback within five minutes." Gradle built an entire product category, developer productivity engineering, around instrumenting builds the way you'd instrument a fleet of servers. And the whole platform-engineering movement rests on the premise that developers are internal customers of a product that can be up or down. So the thesis isn't novel. What's changed is the economics that make it urgent — and Orr's post is a symptom of that shift, not the cause. Two forces just raised the stakes First, the security story settled the argument about whether CI is production-critical. The January 2023 CircleCI breach https://circleci.com/blog/jan-4-2023-incident-report started with malware on one engineer's laptop and ended with customers rotating every secret they'd ever stored in the platform — API tokens, SSH keys, cloud credentials. Your pipeline holds keys to everything you run. An attacker doesn't distinguish between "internal tooling" and "production"; neither should your threat model or your reliability posture. Second — and this is the 2026-specific part — coding agents changed the load profile. When your CI's primary customer was a human who kicked off a build and went to get coffee, a 40-minute queue was an annoyance. When a meaningful fraction of your commits come from agents that iterate against CI in tight loops, the pipeline becomes the rate limiter on your entire engineering output, around the clock. Humans route around a broken QA environment; agents just stall or, worse, thrash. If you're spending real money on agentic development, pipeline downtime is now directly metered in dollars, which is exactly the visibility problem HN commenters complained about "if a problem never manifests, was it real?" solving itself. What adopting this actually looks like The failure mode is declaring "CI is production now " and changing nothing except adding pager noise. The useful version is narrower and boring: Pick three or four SLIs and publish them. The ones that matter almost everywhere: time from push to CI start queue health , p95 pipeline duration on the main merge path, time-to-green after main goes red, and flake rate on the blocking test suite. All four fall out of data your CI system already has. DORA's https://dora.dev/ lead-time-for-changes metric is the umbrella number, but it's too coarse to debug with; these are its components. Give the pipeline an owner and an error budget. Not "the last person who touched the Jenkinsfile." An owning team, with an agreement that when the flake budget or queue-time budget is blown, that team's feature work pauses in favor of reliability work — the standard SRE bargain, applied internally. This is also the honest answer to the HN commenter who noted the real problem is convincing leadership to fund invisible prevention: an SLO dashboard converts prevention from a vibe into a number that's red or green. Define severity levels that respect the clock. Here's where the "everything is production" crowd overreaches. A broken main branch or a dead CI system is a sev-2 during working hours — full-attention response, incident channel, post-incident review. It is almost never a page at 3 a.m., because nobody's deploy is blocked at 3 a.m. Unless, per the above, your agents work nights — then you get to make that call explicitly instead of by default. The semantic objection on HN — "my IDE breaking is a development system failure, not a production one" — is right about the words and wrong about the practices. You don't need the label; you need the SLO, the owner, and the review. Treat pipeline changes like production changes. Staged rollout of CI config, the ability to roll back a runner image, monitoring on the monitors. The teams that get burned worst are the ones where a global pipeline change ships to 100% of builds instantly — a deployment practice they'd never tolerate for customer traffic. Who this lands on Platform and DevEx teams should be printing Orr's post out, because it's budget ammunition: the argument for headcount has always been "developer time is expensive," and now there's a second, sharper one — "our agents are idle when CI is down, and here's the invoice." The losers are organizations running CI as unowned leftover ops, because the gap between them and teams with pipeline SLOs compounds: every reliability improvement to the pipeline multiplies across every human and every agent that uses it. The pipeline was always a production system. It just took the machines becoming its heaviest users for the humans to start treating it like one. Sources & further reading - The development pipeline is a production system https://sundry.jerryorr.com/2026/07/31/development-pipeline-is-a-production-system — sundry.jerryorr.com - HN discussion: The development pipeline is a production system https://news.ycombinator.com/item?id=49130726 — news.ycombinator.com - Applying SRE principles to CI/CD https://buildkite.com/resources/blog/applying-sre-principles-to-cicd/ — buildkite.com - Flaky Tests at Google and How We Mitigate Them https://testing.googleblog.com/2016/05/flaky-tests-at-google-and-how-we.html — testing.googleblog.com - CircleCI incident report for January 4, 2023 security incident https://circleci.com/blog/jan-4-2023-incident-report — circleci.com Lenn Voss https://sourcefeed.dev/u/lennart voss · Cloud & Infrastructure Writer Lenn writes about cloud platforms, Kubernetes internals, and the infrastructure decisions that quietly make or break engineering organizations. Based in Berlin's vibrant tech scene, they have a talent for turning dense platform-engineering topics into prose that people actually finish reading. Discussion 0 No comments yet Be the first to weigh in.