cd /news/developer-tools/ci-cd-is-a-production-dependency-bre… · home › topics › developer-tools › article
[ARTICLE · art-173305] src=dev.to ↗ pub= topic=developer-tools verified=true sentiment=· neutral

CI/CD Is a Production Dependency: Break-Glass Deploy Paths for the Next CI/CD Outage

A developer outlined a break-glass deploy strategy for surviving CI/CD outages, prompted by GitHub's November 11, 2026 incident that took down Actions, Pages and Copilot for roughly 33 minutes. The approach requires a runner the team controls, pre-built artifacts mirrored in an owned registry, break-glass credentials independent of the CI provider's OIDC tokens, and a runbook that has actually been exercised, with deploys pinned by image digest rather than tag.

recommendation

Build and rehearse a break-glass deploy that assumes a separate MFA-gated role and ships a pre-scanned image by digest, so you can roll back or hotfix when your CI provider is down.

more in playbook: Running AI in production
by read4 min views1 publishedNov 20, 2026

On November 11, 2026, at 20:48 UTC, GitHub opened an incident titled "Impact across multiple GitHub products including Actions, Pages, and Copilot products". The status page marked Actions, Pages and Copilot as a major outage and Webhooks as degraded. GitHub reported mitigation at 21:07 UTC and resolved the incident at 21:21 UTC, after the backlog of queued Actions jobs was processed.

That is a short window. But if your only way to ship was a GitHub Actions workflow, you could not deploy during it. Not a hotfix, not a rollback, not a config change. That is the real lesson of any CI/CD outage: your pipeline is a production dependency, and it deserves the same failover thinking as your database. The answer is a break-glass deploy path.

I posted a shorter take on X. This is the longer version.

We spend a lot of effort on multi-AZ databases and redundant load balancers. Then we route every change to production through one hosted service, triggered by one webhook system, authenticated by tokens that service issues.

Look at what a typical hosted pipeline depends on:

If any one of these fails, deploys stop. Most of the time that is fine, because you can wait. The problem is the day you cannot wait: an active incident, a bad release that needs a rollback, or an expiring certificate.

A break-glass deploy is a documented, tested, audited way to ship a known-good artifact to production when the normal pipeline is unavailable. It is not a back door. It should be slower and more controlled than the normal path, and every use should be reviewed afterwards.

In my view it needs four parts.

A runner you control. A self-hosted runner, or a secondary CI system, that can run the same deploy step. Self-hosted runners registered to GitHub still need GitHub to dispatch jobs, so they protect you from hosted runner capacity problems, not from a full Actions outage. For that, you need a deploy you can start from a hardened bastion or a second CI system with no dependency on the first.

Artifacts that already exist. Break-glass is for deploying something already built and scanned, not for building from a laptop. Keep release artifacts in storage you own, and mirror the container images you deploy into your own registry, so a deploy never pulls from a public registry or the CI provider's cache.

Credentials that do not depend on the CI provider. This is the part teams miss. OIDC federation between GitHub Actions and your cloud is a good default because it removes long-lived keys. But that trust only works when Actions is issuing tokens. If Actions is down, that path is gone. Keep a separate, tightly scoped break-glass role: it can only deploy, it needs phishing-resistant MFA and approval from a second person, sessions are short, and every assumption triggers an alert.

A runbook people have actually used. A runbook nobody has run is a guess.

Here is the shape I like. Every value is a placeholder.


aws sts assume-role \
  --role-arn "arn:aws:iam::<ACCOUNT_ID>:role/<BREAK_GLASS_DEPLOY_ROLE>" \
  --role-session-name "breakglass-<INCIDENT_ID>-<YOUR_NAME>" \
  --serial-number "<MFA_DEVICE_ARN>" --token-code "<MFA_CODE>" \
  --duration-seconds 900

kubectl --context <PROD_CONTEXT> -n <NAMESPACE> set image \
  deployment/<APP> <CONTAINER>=<REGISTRY_MIRROR>/<APP>@sha256:<DIGEST>

kubectl --context <PROD_CONTEXT> -n <NAMESPACE> rollout status deployment/<APP>

Pinning by digest matters. Under pressure, nobody should be guessing which latest they just shipped.

The GitHub incident was resolved in well under an hour. Next time it might be longer, or it might be a different provider. The teams that stay calm are the ones that already know their fallback works because they ran it last month.

Write the runbook, stand up the runner, mirror the images, scope the role, and drill it. Then a CI/CD outage becomes an inconvenience instead of a blocker. I write more about platform reliability and security on my site.

I lead platform engineering for fintech. See more of my work or how we can work together at Nazmul Ahasan.

── more in #developer-tools 4 stories · sorted by recency
── more on @github 3 stories trending now
sponsored brought to you by zahid.host 4,200+ EU-deployed projects
reading about agents? ship yours in a single git push.

Run your AI side-project on zahid.host

EU-based hosting, git-push deploys, automatic HTTPS, no cold starts. Free tier with a custom domain — perfect for shipping the agent you just read about.

$git push zahid main
→ Live at https://your-agent.zahid.host ✓
Get free account → Pricing
from €0/mo · no card required
LIVE [news/ci-cd-is-a-productio…] indexed:0 read:4min 2026-11-20 · —