On November 11, 2026, at 20:48 UTC, GitHub opened an incident titled "Impact across multiple GitHub products including Actions, Pages, and Copilot products". The status page marked Actions, Pages and Copilot as a major outage and Webhooks as degraded. GitHub reported mitigation at 21:07 UTC and resolved the incident at 21:21 UTC, after the backlog of queued Actions jobs was processed.
That is a short window. But if your only way to ship was a GitHub Actions workflow, you could not deploy during it. Not a hotfix, not a rollback, not a config change. That is the real lesson of any CI/CD outage: your pipeline is a production dependency, and it deserves the same failover thinking as your database. The answer is a break-glass deploy path.
I posted a shorter take on X. This is the longer version.
We spend a lot of effort on multi-AZ databases and redundant load balancers. Then we route every change to production through one hosted service, triggered by one webhook system, authenticated by tokens that service issues.
Look at what a typical hosted pipeline depends on:
If any one of these fails, deploys stop. Most of the time that is fine, because you can wait. The problem is the day you cannot wait: an active incident, a bad release that needs a rollback, or an expiring certificate.
A break-glass deploy is a documented, tested, audited way to ship a known-good artifact to production when the normal pipeline is unavailable. It is not a back door. It should be slower and more controlled than the normal path, and every use should be reviewed afterwards.
In my view it needs four parts.
A runner you control. A self-hosted runner, or a secondary CI system, that can run the same deploy step. Self-hosted runners registered to GitHub still need GitHub to dispatch jobs, so they protect you from hosted runner capacity problems, not from a full Actions outage. For that, you need a deploy you can start from a hardened bastion or a second CI system with no dependency on the first.
Artifacts that already exist. Break-glass is for deploying something already built and scanned, not for building from a laptop. Keep release artifacts in storage you own, and mirror the container images you deploy into your own registry, so a deploy never pulls from a public registry or the CI provider's cache.
Credentials that do not depend on the CI provider. This is the part teams miss. OIDC federation between GitHub Actions and your cloud is a good default because it removes long-lived keys. But that trust only works when Actions is issuing tokens. If Actions is down, that path is gone. Keep a separate, tightly scoped break-glass role: it can only deploy, it needs phishing-resistant MFA and approval from a second person, sessions are short, and every assumption triggers an alert.
A runbook people have actually used. A runbook nobody has run is a guess.
Here is the shape I like. Every value is a placeholder.
aws sts assume-role \
--role-arn "arn:aws:iam::<ACCOUNT_ID>:role/<BREAK_GLASS_DEPLOY_ROLE>" \
--role-session-name "breakglass-<INCIDENT_ID>-<YOUR_NAME>" \
--serial-number "<MFA_DEVICE_ARN>" --token-code "<MFA_CODE>" \
--duration-seconds 900
kubectl --context <PROD_CONTEXT> -n <NAMESPACE> set image \
deployment/<APP> <CONTAINER>=<REGISTRY_MIRROR>/<APP>@sha256:<DIGEST>
kubectl --context <PROD_CONTEXT> -n <NAMESPACE> rollout status deployment/<APP>
Pinning by digest matters. Under pressure, nobody should be guessing which latest they just shipped.
The GitHub incident was resolved in well under an hour. Next time it might be longer, or it might be a different provider. The teams that stay calm are the ones that already know their fallback works because they ran it last month.
Write the runbook, stand up the runner, mirror the images, scope the role, and drill it. Then a CI/CD outage becomes an inconvenience instead of a blocker. I write more about platform reliability and security on my site.
I lead platform engineering for fintech. See more of my work or how we can work together at Nazmul Ahasan.