{"slug": "ci-cd-is-a-production-dependency-break-glass-deploy-paths-for-the-next-ci-cd", "title": "CI/CD Is a Production Dependency: Break-Glass Deploy Paths for the Next CI/CD Outage", "summary": "A developer outlined a break-glass deploy strategy for surviving CI/CD outages, prompted by GitHub's November 11, 2026 incident that took down Actions, Pages and Copilot for roughly 33 minutes. The approach requires a runner the team controls, pre-built artifacts mirrored in an owned registry, break-glass credentials independent of the CI provider's OIDC tokens, and a runbook that has actually been exercised, with deploys pinned by image digest rather than tag.", "body_md": "On November 11, 2026, at 20:48 UTC, GitHub opened an incident titled [\"Impact across multiple GitHub products including Actions, Pages, and Copilot products\"](https://www.githubstatus.com/incidents/dvy0t3mvh1lg). The status page marked Actions, Pages and Copilot as a major outage and Webhooks as degraded. GitHub reported mitigation at 21:07 UTC and resolved the incident at 21:21 UTC, after the backlog of queued Actions jobs was processed.\n\nThat is a short window. But if your only way to ship was a GitHub Actions workflow, you could not deploy during it. Not a hotfix, not a rollback, not a config change. That is the real lesson of any CI/CD outage: your pipeline is a production dependency, and it deserves the same failover thinking as your database. The answer is a **break-glass deploy** path.\n\nI posted a [shorter take on X](https://x.com/nrejve/status/2120923202797080790). This is the longer version.\n\nWe spend a lot of effort on multi-AZ databases and redundant load balancers. Then we route every change to production through one hosted service, triggered by one webhook system, authenticated by tokens that service issues.\n\nLook at what a typical hosted pipeline depends on:\n\nIf any one of these fails, deploys stop. Most of the time that is fine, because you can wait. The problem is the day you cannot wait: an active incident, a bad release that needs a rollback, or an expiring certificate.\n\nA break-glass deploy is a documented, tested, audited way to ship a known-good artifact to production when the normal pipeline is unavailable. It is not a back door. It should be slower and more controlled than the normal path, and every use should be reviewed afterwards.\n\nIn my view it needs four parts.\n\n**A runner you control.** A self-hosted runner, or a secondary CI system, that can run the same deploy step. Self-hosted runners registered to GitHub still need GitHub to dispatch jobs, so they protect you from hosted runner capacity problems, not from a full Actions outage. For that, you need a deploy you can start from a hardened bastion or a second CI system with no dependency on the first.\n\n**Artifacts that already exist.** Break-glass is for deploying something already built and scanned, not for building from a laptop. Keep release artifacts in storage you own, and mirror the container images you deploy into your own registry, so a deploy never pulls from a public registry or the CI provider's cache.\n\n**Credentials that do not depend on the CI provider.** This is the part teams miss. OIDC federation between GitHub Actions and your cloud is a good default because it removes long-lived keys. But that trust only works when Actions is issuing tokens. If Actions is down, that path is gone. Keep a separate, tightly scoped break-glass role: it can only deploy, it needs phishing-resistant MFA and approval from a second person, sessions are short, and every assumption triggers an alert.\n\n**A runbook people have actually used.** A runbook nobody has run is a guess.\n\nHere is the shape I like. Every value is a placeholder.\n\n```\n# 0. Declare it. Open an incident and record who approved break-glass.\n\n# 1. Assume the break-glass role (MFA required, short session, alerts on use)\naws sts assume-role \\\n  --role-arn \"arn:aws:iam::<ACCOUNT_ID>:role/<BREAK_GLASS_DEPLOY_ROLE>\" \\\n  --role-session-name \"breakglass-<INCIDENT_ID>-<YOUR_NAME>\" \\\n  --serial-number \"<MFA_DEVICE_ARN>\" --token-code \"<MFA_CODE>\" \\\n  --duration-seconds 900\n\n# 2. Deploy a known-good, already-scanned image by digest, never by tag\nkubectl --context <PROD_CONTEXT> -n <NAMESPACE> set image \\\n  deployment/<APP> <CONTAINER>=<REGISTRY_MIRROR>/<APP>@sha256:<DIGEST>\n\n# 3. Verify, then record what changed in the incident ticket\nkubectl --context <PROD_CONTEXT> -n <NAMESPACE> rollout status deployment/<APP>\n```\n\nPinning by digest matters. Under pressure, nobody should be guessing which `latest` they just shipped.\n\nThe GitHub incident was resolved in well under an hour. Next time it might be longer, or it might be a different provider. The teams that stay calm are the ones that already know their fallback works because they ran it last month.\n\nWrite the runbook, stand up the runner, mirror the images, scope the role, and drill it. Then a CI/CD outage becomes an inconvenience instead of a blocker. I write more about platform reliability and security on [my site](https://nazmul-ahasan.pro.bd/).\n\nI lead platform engineering for fintech. See more of my work or how we can work together at [Nazmul Ahasan](https://nazmul-ahasan.pro.bd/work-with-me/).", "url": "https://wpnews.pro/news/ci-cd-is-a-production-dependency-break-glass-deploy-paths-for-the-next-ci-cd", "canonical_source": "https://dev.to/nrejve/cicd-is-a-production-dependency-break-glass-deploy-paths-for-the-next-cicd-outage-49a8", "published_at": "2026-11-20 15:04:22+00:00", "updated_at": "2026-11-20 15:15:23.752256+00:00", "lang": "en", "topics": ["developer-tools", "mlops"], "entities": ["GitHub", "GitHub Actions", "GitHub Copilot", "GitHub Pages", "AWS"], "also_reported_by": [], "takeaway": "Keep a break-glass deploy path with a self-controlled runner, artifacts mirrored in your own registry, and a separate short-session MFA-gated deploy role, since GitHub Actions OIDC tokens vanish when Actions is down; deploy by image digest, not tag.", "recommendation": {"advice": "Build and rehearse a break-glass deploy that assumes a separate MFA-gated role and ships a pre-scanned image by digest, so you can roll back or hotfix when your CI provider is down.", "playbook": "production-ops", "playbook_url": "https://wpnews.pro/playbooks/production-ops"}, "alternates": {"html": "https://wpnews.pro/news/ci-cd-is-a-production-dependency-break-glass-deploy-paths-for-the-next-ci-cd", "markdown": "https://wpnews.pro/news/ci-cd-is-a-production-dependency-break-glass-deploy-paths-for-the-next-ci-cd.md", "text": "https://wpnews.pro/news/ci-cd-is-a-production-dependency-break-glass-deploy-paths-for-the-next-ci-cd.txt", "jsonld": "https://wpnews.pro/news/ci-cd-is-a-production-dependency-break-glass-deploy-paths-for-the-next-ci-cd.jsonld"}}