Frontend CI/CD in the age of AI - Part 2: Deployments Neciu Dan, a technical co-founder and Staff Software Engineer, outlines a deployment pipeline for frontend CI/CD in the age of AI, emphasizing canary deployments and rapid rollbacks to counter the increased delivery instability reported in DORA's 2025 report, which links higher AI adoption to more production failures. The proposed pipeline uses a merge queue, a single build artifact, and staged traffic shifts with smoke tests and automated rollbacks, aiming to reduce the hours-long rebuilds typical of all-or-nothing deployments. · devops /category/devops · 37 min read Frontend CI/CD in the age of AI - Part 2: Deployments Agents write code that passes every test and still breaks in production. Here is how to ship it to a slice of your users first, and how to take it back in seconds when it goes wrong. Neciu Dan Hi there, it's Dan, a technical co-founder of an ed-tech startup, host of Señors at Scale - a podcast for Senior Engineers, Organizer of ReactJS Barcelona meetup, international speaker and Staff Software Engineer, I'm here to share insights on combining technology and education to solve real problems. I write about startup challenges, tech innovations, and the Frontend Development. Subscribe to join me on this journey of transforming education through technology. Want to discuss Tech, Frontend or Startup life? Let's connect. https://www.linkedin.com/in/neciudan/ Part 1 https://neciudan.dev/ci-cd-in-the-age-of-ai-part-1 covered the Continuous Integration part, where the goal was to run the pipeline less cheaper and run it faster, because agents produce more pull requests than a normal pipeline can absorb. On the other hand, while AI has gotten better and better at coding and reviewing code, it still is missing context and product sense, and as a result, it makes mistakes. A lot of mistakes. And this is an industry problem. If you look carefully, reliability is down across the board. DORA’s 2025 report says that teams with higher AI adoption show both higher delivery throughput and higher delivery instability, resulting in more changes failing in production and more unplanned work to fix them. This makes our deployment part that much more important. We need to either deploy to a subset of our users to ensure it doesn’t cause outages, and revert quickly if something goes wrong. Unfortunately, that is not how we are doing things. Most of the companies I worked with typically use an all-or-nothing approach to deployments, and reverting often requires rebuilding the entire CI pipeline, which can take hours. Here is how our ideal pipeline looks like 👇 A pull request merges into main only through a merge queue, where the full integration suite runs against the latest code. We do this because in the Part 1 article, we only checked what was touched or impacted by the PR code. The merge produces a single build artifact, which we use for deployment and later serve in production without being rebuilt, so promoting and rolling back both come down to where the traffic is pointing. That artifact goes out to a piece of your traffic, with a smoke test hitting it the moment it lands. Five minutes later, something looks for catastrophic movement, and an hour after that, a slower comparison against the previous release decides whether the artifact is ready for 100% traffic. A bad result rolls it back, posts the offending pull request to Slack, opens a revert branch, and freezes merging until somebody closes the incident. In this article, we are building this pipeline or parts of it . Let’s go. Canaries British coal mines carried canaries underground until 1986. A canary breathes through a one-way system with air sacs feeding the lungs, and it burns oxygen fast enough that carbon monoxide fills it well before anyone holding the cage feels a thing. A bird that stopped singing and swayed on its perch bought the crew a few minutes to climb out. Later cages came with a small oxygen bottle attached, so you could seal the bird in and revive it on the way up. Deployments applied the same concept. A small, known group goes into the unverified place ahead of everybody else, and you watch them while the rest of your traffic carries on where it was. Your five percent of traffic is the canary, and when you have a broken release, they are paying the price. To mitigate for randomness and not affect important users, some companies apply this pattern to geo-location traffic, testing in low-impact countries The workflow we start from Most teams have something close to this, which builds the app and ships it in one go. .github/workflows/deploy.yml name: Deploy on: push: branches: main jobs: deploy: runs-on: ubuntu-latest steps: - uses: actions/checkout@v4 - uses: actions/setup-node@v4 with: node-version: 22 cache: 'npm' - run: npm ci - run: npm run build - run: npx vercel deploy --prod --token="$VERCEL TOKEN" env: VERCEL TOKEN: ${{ secrets.VERCEL TOKEN }} That last step is the only line that changes from platform to platform, and the moment it finishes, every user gets the new code. Our pipeline, regardless of the platform we are deploying to, will use these concepts and I will refrence them all through the article: deploy ships the build and gives it a small slice of traffic smoke hits the new version before anybody else does with some tests soak waits, then reads your error rate split by version promote gives the build all the traffic abort removes the traffic from the build and gives it to the previous version hold posts to Slack when the numbers are too thin to call watch-15 and watch-60 keep checking after promotion rollback puts the previous build back when they find something The common setup Every job below starts the same way: .github/actions/setup/action.yml name: Setup description: Node and dependencies runs: using: composite steps: - uses: actions/setup-node@v4 with: node-version: 22 cache: 'npm' - run: npm ci shell: bash Composite actions need shell: on every run step, and the action lives in your repository, so actions/checkout has to run before you can call it. The environments The soak-and-watch jobs need to pause. If you use a step that does sleep 1800 it will work but it will also occupy a runner for half an hour, billed to you on every merge. GitHub environments do it for free. A job targeting an environment with a wait timer sits waiting before it starts, outside a runner. In the Settings dashboard, then Environments, then New environment, tick Wait timer, and enter the minutes, with 43200 as the ceiling. All four at once from the CLI: REPO=my-org/my-app gh api -X PUT "repos/$REPO/environments/canary-soak" -F wait timer=60 gh api -X PUT "repos/$REPO/environments/watch-15min" -F wait timer=15 gh api -X PUT "repos/$REPO/environments/watch-45min" -F wait timer=45 gh api -X PUT "repos/$REPO/environments/geo-soak" -F wait timer=60 Telling Sentry which release is which The soak job compares two releases for bugs and issues , which only works if your app reports which one it is. js // vite.config.ts const release = process.env.VERCEL GIT COMMIT SHA ?? // Vercel builds on its own machines process.env.COMMIT REF ?? // so does Netlify process.env.GITHUB SHA ?? // Cloudflare, Cloud Run, and ECS build in CI 'local'; export default defineConfig { define: { 'import.meta.env.RELEASE': JSON.stringify release }, } ; Vercel and Netlify build remotely, where GITHUB SHA does not exist, so a config that reads only that variable tags every release as local , and the soak job compares two things that are not there. All three variables hold the full commit SHA, which is what github.sha gives the workflow. Sentry.init { dsn: import.meta.env.VITE SENTRY DSN, release: import.meta.env.RELEASE, } ; Without this, every query returns the same blended number for both versions, and the check can never tell them apart. The script that decides We neeed to compare releases and for that we need to run a script. The script can return hold , promote , and abort . At five percent traffic on Tuesday early afternoon, half an hour might produce forty sessions, and forty sessions say nothing about whether your crash rate moved. Promoting on numbers that thin is guessing, and aborting throws away a deploy that was probably fine. js // scripts/canary-health.mjs import { appendFileSync } from 'node:fs'; const { SENTRY TOKEN, SENTRY ORG, SENTRY PROJECT ID, CANARY RELEASE, BASELINE RELEASE, MIN SESSIONS = 200 } = process.env; const MAX DROP POINTS = 0.5; // crash-free rate, in percentage points // Sentry declares this metric as crash free rate@ratio , so the API hands // back 0..1 even though the UI shows you 99.98%. Converting on the value // rather than assuming the scale costs nothing and survives Sentry changing // its mind. const points = rate = rate === null || rate === undefined || Number.isNaN rate ? null : rate <= 1 ? rate 100 : rate; async function health release { if release return { rate: null, sessions: 0 }; const url = new URL https://sentry.io/api/0/organizations/${SENTRY ORG}/sessions/ ; url.searchParams.append 'field', 'crash free rate session ' ; url.searchParams.append 'field', 'sum session ' ; url.searchParams.set 'project', SENTRY PROJECT ID ; url.searchParams.set 'statsPeriod', '1h' ; // query= rather than groupBy=release, which has a habit of // returning zeroed buckets for individual releases url.searchParams.set 'query', release:"${release}" ; const response = await fetch url, { headers: { Authorization: Bearer ${SENTRY TOKEN} }, // Without this, a hung Sentry holds the runner to the six-hour job ceiling signal: AbortSignal.timeout 20 000 , } ; if response.ok throw new Error Sentry returned ${response.status} ; const totals = await response.json .groups?. 0 ?.totals; if totals return { rate: null, sessions: 0 }; return { rate: totals 'crash free rate session ' ?? null, sessions: totals 'sum session ' ?? 0, }; } function decide decision, why { console.log ${decision}: ${why} ; // Running this by hand to debug a stuck rollout must not crash if process.env.GITHUB OUTPUT { appendFileSync process.env.GITHUB OUTPUT, decision=${decision}\n ; } process.exit 0 ; } try { const canary = await health CANARY RELEASE ; const baseline = await health BASELINE RELEASE ; if canary.sessions < Number MIN SESSIONS { decide 'hold', only ${canary.sessions} sessions on the canary so far ; } const canaryRate = points canary.rate ; const baselineRate = points baseline.rate ; // undefined slips past a === null check, makes the subtraction NaN, and // NaN fails every comparison — which reads as "no drop" and promotes a // release nobody measured. A baseline with no sessions is not a baseline. if canaryRate === null || baselineRate === null || baseline.sessions === 0 { decide 'hold', 'no session data for one of the two releases' ; } // Round before comparing, so the number in the log is the number // the decision was made on const drop = Number baselineRate - canaryRate .toFixed 2 ; if drop MAX DROP POINTS { decide 'abort', crash-free rate down ${drop.toFixed 2 } points ; } decide 'promote', crash-free rate within ${MAX DROP POINTS} points ; } catch error { // Sentry being unreachable is not evidence that your deploy is bad decide 'hold', health check failed: ${error.message} ; } A hold parks the canary where it is, with production still mostly on the old build, which is a safe place to leave something while you look at it yourself. But be aware that the endpoint buckets by the hour and refuses a window shorter than one, so a 30-minute soak reads 60 minutes of data, including traffic from before your deploy existed. Let it soak for an hour, or swap the query for a count of issues first seen since the deployment timestamp. Until this point everything can be reused regardless of deployment platform. Let’s go into specifics. You can skip to your platform of choice or the end where we build something independend. Vercel We need to change two settings to get canary releases to work on Vercel. Skew Protection. Go to Settings, then Advanced, then switch it on and set Maximum Age. Having two versions live at once means that one browser can load HTML from the old version and JavaScript from the new, resulting in a blank screen. This pins each session to whichever deployment it first loaded. The stages. Rolling Releases needs a Pro or Enterprise plan, and the percentages are fixed before the rollout as configuration. vercel rolling-release configure --cfg '{ "enabled": true, "advancementType": "manual-approval", "canaryResponseHeader": true, "stages": { "targetPercentage": 5 }, { "targetPercentage": 25 }, { "targetPercentage": 60 }, { "targetPercentage": 100 } }' Deploying to five percent instead of everyone The baseline workflow ended at vercel deploy --prod . Two lines change that into a canary. .github/workflows/deploy.yml name: Deploy on: push: branches: main One rollout at a time. Two mergers of racing produce two canaries. GitHub keeps only one pending run per group, so a third merge arriving mid-rollout cancels the second. One at a time is guaranteed; every merge reaching production is not. concurrency: group: production-deploy cancel-in-progress: false permissions: contents: read env: VERCEL TOKEN: ${{ secrets.VERCEL TOKEN }} jobs: deploy: runs-on: ubuntu-latest outputs: url: ${{ steps.deploy.outputs.url }} baseline: ${{ steps.baseline.outputs.id }} steps: - uses: actions/checkout@v4 - uses: ./.github/actions/setup What production is running right now, so the soak job has something to compare against. The whole rollingRelease object is null until a deployment has happened, so your first ever run gets "null" here and the health check holds rather than promoting. There is a sibling canaryDeployment.id if you would rather name the canary that way than by commit SHA. - id: baseline run: | ID=$ npx vercel rolling-release fetch --token="$VERCEL TOKEN" \ | jq -r '.rollingRelease.currentDeployment.id' echo "id=$ID" "$GITHUB OUTPUT" - id: deploy run: | URL=$ npx vercel deploy --prod --token="$VERCEL TOKEN" echo "url=$URL" "$GITHUB OUTPUT" A start that fails leaves the deployment live on all the traffic, with no smoke test run and no abort job watching it if npx vercel rolling-release start --dpl="$URL" --token="$VERCEL TOKEN" --yes; then npx vercel rolling-release abort --dpl="$URL" --token="$VERCEL TOKEN" --yes || true exit 1 fi rolling-release start is doing all the work in that step. It leaves the deployment sitting at stage zero on five percent, where, without it, the deployment would have taken everything. That job doesn’t include npm run build , because Vercel builds on its own infrastructure when you run vercel deploy . Hitting it before your users do smoke: needs: deploy runs-on: ubuntu-latest timeout-minutes: 5 steps: - uses: actions/checkout@v4 - uses: ./.github/actions/setup - run: npx playwright install --with-deps chromium - run: npx playwright test e2e/smoke --project=chromium env: PLAYWRIGHT BASE URL: ${{ needs.deploy.outputs.url }} Waiting, then deciding soak: needs: deploy, smoke runs-on: ubuntu-latest environment: canary-soak the 60-minute wait lives here outputs: decision: ${{ steps.check.outputs.decision }} steps: - uses: actions/checkout@v4 - uses: ./.github/actions/setup - id: check run: node scripts/canary-health.mjs env: SENTRY TOKEN: ${{ secrets.SENTRY TOKEN }} SENTRY ORG: my-org SENTRY PROJECT ID: '4504000000000000' CANARY RELEASE: ${{ github.sha }} BASELINE RELEASE: ${{ needs.deploy.outputs.baseline }} Taking the rest, or giving it back promote: needs: deploy, soak if: needs.soak.outputs.decision == 'promote' runs-on: ubuntu-latest steps: - uses: actions/checkout@v4 - uses: ./.github/actions/setup complete skips the remaining stages and takes 100% - run: npx vercel rolling-release complete --dpl='${{ needs.deploy.outputs.url }}' --token="$VERCEL TOKEN" --yes abort: needs: deploy, smoke, soak if: always && needs.smoke.result == 'failure' || needs.soak.outputs.decision == 'abort' runs-on: ubuntu-latest steps: - uses: actions/checkout@v4 - uses: ./.github/actions/setup - run: npx vercel rolling-release abort --dpl='${{ needs.deploy.outputs.url }}' --token="$VERCEL TOKEN" --yes - run: | curl -sS -X POST -H 'Content-type: application/json' \ --data "{\"text\": \"Canary aborted. Production is back on the previous build.\"}" \ "${{ secrets.SLACK WEBHOOK }}" A hold decision matches neither condition, so the canary stays at five percent and nothing else runs. The full file below adds a third job on decision == 'hold' that posts to Slack, because otherwise you find rollouts parked there days later. Rolling back after promotion A release that behaves at 5% for an hour can still fail at full traffic. We want to make sure we can always do a fast and safe rollback. watch-15: needs: deploy, promote runs-on: ubuntu-latest environment: watch-15min outputs: decision: ${{ steps.check.outputs.decision }} steps: - uses: actions/checkout@v4 - uses: ./.github/actions/setup - id: check run: node scripts/canary-health.mjs env: SENTRY TOKEN: ${{ secrets.SENTRY TOKEN }} SENTRY ORG: my-org SENTRY PROJECT ID: '4504000000000000' CANARY RELEASE: ${{ github.sha }} BASELINE RELEASE: ${{ needs.deploy.outputs.baseline }} watch-60: needs: deploy, promote, watch-15 if: needs.watch-15.outputs.decision = 'abort' runs-on: ubuntu-latest environment: watch-45min runs after watch-15, so 15 + 45 = an hour in outputs: decision: ${{ steps.check.outputs.decision }} steps: - uses: actions/checkout@v4 - uses: ./.github/actions/setup - id: check run: node scripts/canary-health.mjs env: SENTRY TOKEN: ${{ secrets.SENTRY TOKEN }} SENTRY ORG: my-org SENTRY PROJECT ID: '4504000000000000' CANARY RELEASE: ${{ github.sha }} BASELINE RELEASE: ${{ needs.deploy.outputs.baseline }} rollback: needs: deploy, watch-15, watch-60 if: always && needs.watch-15.outputs.decision == 'abort' || needs.watch-60.outputs.decision == 'abort' runs-on: ubuntu-latest steps: - uses: actions/checkout@v4 - uses: ./.github/actions/setup - run: npx vercel rollback '${{ needs.deploy.outputs.baseline }}' --token="$VERCEL TOKEN" - run: | curl -sS -X POST -H 'Content-type: application/json' \ --data "{\"text\": \"Rolled back after promotion.\"}" \ "${{ secrets.SLACK WEBHOOK }}" The previous build never went anywhere, so rollback points the domain at something that already exists and finishes in seconds. Sessions you pinned stay where they are, though. Skew Protection keeps them loading assets from the deployment they landed on until Maximum Age expires, so a rollback catches everybody arriving fresh while open tabs sit put. The whole file Every fragment above, plus the hold job the soak decision needs, in one file. .github/workflows/deploy.yml name: Deploy on: push: branches: main concurrency: group: production-deploy cancel-in-progress: false permissions: contents: read env: VERCEL TOKEN: ${{ secrets.VERCEL TOKEN }} SENTRY TOKEN: ${{ secrets.SENTRY TOKEN }} SENTRY ORG: my-org SENTRY PROJECT ID: '4504000000000000' jobs: deploy: runs-on: ubuntu-latest outputs: url: ${{ steps.deploy.outputs.url }} baseline: ${{ steps.baseline.outputs.id }} steps: - uses: actions/checkout@v4 - uses: ./.github/actions/setup - id: baseline run: | ID=$ npx vercel rolling-release fetch --token="$VERCEL TOKEN" \ | jq -r '.rollingRelease.currentDeployment.id' echo "id=$ID" "$GITHUB OUTPUT" - id: deploy run: | URL=$ npx vercel deploy --prod --token="$VERCEL TOKEN" echo "url=$URL" "$GITHUB OUTPUT" A start that fails leaves the deployment live on all the traffic, with no smoke test run and no abort job watching it if npx vercel rolling-release start --dpl="$URL" --token="$VERCEL TOKEN" --yes; then npx vercel rolling-release abort --dpl="$URL" --token="$VERCEL TOKEN" --yes || true exit 1 fi smoke: needs: deploy runs-on: ubuntu-latest timeout-minutes: 5 steps: - uses: actions/checkout@v4 - uses: ./.github/actions/setup - run: npx playwright install --with-deps chromium - run: npx playwright test e2e/smoke --project=chromium env: PLAYWRIGHT BASE URL: ${{ needs.deploy.outputs.url }} soak: needs: deploy, smoke runs-on: ubuntu-latest environment: canary-soak outputs: decision: ${{ steps.check.outputs.decision }} steps: - uses: actions/checkout@v4 - uses: ./.github/actions/setup - id: check run: node scripts/canary-health.mjs env: CANARY RELEASE: ${{ github.sha }} BASELINE RELEASE: ${{ needs.deploy.outputs.baseline }} promote: needs: deploy, soak if: needs.soak.outputs.decision == 'promote' runs-on: ubuntu-latest steps: - uses: actions/checkout@v4 - uses: ./.github/actions/setup - run: npx vercel rolling-release complete --dpl='${{ needs.deploy.outputs.url }}' --token="$VERCEL TOKEN" --yes hold: needs: soak if: needs.soak.outputs.decision == 'hold' runs-on: ubuntu-latest steps: - run: | curl -sS -X POST -H 'Content-type: application/json' \ --data "{\"text\": \"Canary parked at 5%. Not enough data to decide.\"}" \ "${{ secrets.SLACK WEBHOOK }}" abort: needs: deploy, smoke, soak if: always && needs.smoke.result == 'failure' || needs.soak.outputs.decision == 'abort' runs-on: ubuntu-latest steps: - uses: actions/checkout@v4 - uses: ./.github/actions/setup - run: npx vercel rolling-release abort --dpl='${{ needs.deploy.outputs.url }}' --token="$VERCEL TOKEN" --yes - run: | curl -sS -X POST -H 'Content-type: application/json' \ --data "{\"text\": \"Canary aborted. Production is back on the previous build.\"}" \ "${{ secrets.SLACK WEBHOOK }}" watch-15: needs: deploy, promote runs-on: ubuntu-latest environment: watch-15min outputs: decision: ${{ steps.check.outputs.decision }} steps: - uses: actions/checkout@v4 - uses: ./.github/actions/setup - id: check run: node scripts/canary-health.mjs env: CANARY RELEASE: ${{ github.sha }} BASELINE RELEASE: ${{ needs.deploy.outputs.baseline }} watch-60: needs: deploy, promote, watch-15 if: needs.watch-15.outputs.decision = 'abort' runs-on: ubuntu-latest environment: watch-45min runs after watch-15, so 15 + 45 = an hour in outputs: decision: ${{ steps.check.outputs.decision }} steps: - uses: actions/checkout@v4 - uses: ./.github/actions/setup - id: check run: node scripts/canary-health.mjs env: CANARY RELEASE: ${{ github.sha }} BASELINE RELEASE: ${{ needs.deploy.outputs.baseline }} rollback: needs: deploy, watch-15, watch-60 if: always && needs.watch-15.outputs.decision == 'abort' || needs.watch-60.outputs.decision == 'abort' runs-on: ubuntu-latest steps: - uses: actions/checkout@v4 - uses: ./.github/actions/setup - run: npx vercel rollback '${{ needs.deploy.outputs.baseline }}' --token="$VERCEL TOKEN" - run: | curl -sS -X POST -H 'Content-type: application/json' \ --data "{\"text\": \"Rolled back after promotion.\"}" \ "${{ secrets.SLACK WEBHOOK }}" Rolling out to one country Rolling Releases takes a percentage and nothing else, so a country-scoped rollout means routing in middleware and a config value your workflow can move. Create the store once from the dashboard, under Storage, then Edge Config, then Create, and connect it to your project on the same screen. Vercel injects an EDGE CONFIG environment variable for you. js // middleware.ts import { NextResponse, type NextRequest } from 'next/server'; import { geolocation } from '@vercel/functions'; import { get } from '@vercel/edge-config'; export const config = { matcher: '/ ? next/static| next/image|favicon.ico . ' , }; export async function middleware request: NextRequest { const rollout = await get<{ countries: string ; percentage: number; canaryHost: string; } 'canaryRollout' ; const { country } = geolocation request ; if rollout?.countries?.includes country ?? '' { return NextResponse.next ; } // Returning visitors keep whatever they were given const existing = request.cookies.get 'canary' ?.value; const assigned = existing ?? Math.random 100 < rollout.percentage ? 'on' : 'off' ; const url = new URL request.url ; url.host = rollout.canaryHost; const response = assigned === 'on' ? NextResponse.rewrite url : NextResponse.next ; response.cookies.set 'canary', assigned, { maxAge: 86400, path: '/' } ; return response; } The rollout then gets its own workflow, triggered by hand, with the same soak-and-decide loop pointing to the config value rather than a platform percentage. .github/workflows/geo-rollout.yml name: Geo rollout on: workflow dispatch: inputs: countries: description: 'Comma-separated ISO codes, for example PT, IE' required: true concurrency: group: geo-rollout cancel-in-progress: false jobs: start: runs-on: ubuntu-latest outputs: baseline: ${{ steps.baseline.outputs.id }} steps: - uses: actions/checkout@v4 - uses: ./.github/actions/setup Without this, the health check compares against nothing, the arithmetic gives NaN, and every rollout promotes itself - id: baseline run: | ID=$ npx vercel rolling-release fetch --token="$VERCEL TOKEN" \ | jq -r '.rollingRelease.currentDeployment.id' echo "id=$ID" "$GITHUB OUTPUT" - name: 10% of the named countries run: | COUNTRIES=$ echo "${{ inputs.countries }}" | jq -R 'split "," ' npx vercel edge-config items add canaryRollout --value \ "$ jq -nc --argjson c "$COUNTRIES" \ '{countries:$c,percentage:10,canaryHost:"canary.example.com"}' " soak-10: needs: start runs-on: ubuntu-latest environment: geo-soak outputs: decision: ${{ steps.check.outputs.decision }} steps: - uses: actions/checkout@v4 - uses: ./.github/actions/setup - id: check run: node scripts/canary-health.mjs env: SENTRY TOKEN: ${{ secrets.SENTRY TOKEN }} SENTRY ORG: my-org SENTRY PROJECT ID: '4504000000000000' CANARY RELEASE: ${{ github.sha }} BASELINE RELEASE: ${{ needs.start.outputs.baseline }} MIN SESSIONS: 40 ten percent of Portugal is a small number widen: needs: soak-10 if: needs.soak-10.outputs.decision == 'promote' runs-on: ubuntu-latest steps: - uses: actions/checkout@v4 - uses: ./.github/actions/setup - run: | COUNTRIES=$ echo "${{ inputs.countries }}" | jq -R 'split "," ' npx vercel edge-config items add canaryRollout --value \ "$ jq -nc --argjson c "$COUNTRIES" \ '{countries:$c,percentage:50,canaryHost:"canary.example.com"}' " stop: needs: soak-10 if: always && needs.soak-10.outputs.decision == 'abort' runs-on: ubuntu-latest steps: - uses: actions/checkout@v4 - uses: ./.github/actions/setup - run: npx vercel edge-config items add canaryRollout --value '{"percentage":0}' Keep that MIN SESSIONS override. Ten percent of Portugal produces far fewer sessions than ten percent of everything, and the default threshold would return hold on every run. Netlify Netlify has a built-in canary called Split Testing, and you can only use it from the Netlify Dashboard. Turn on branch deploys under Site configuration Build and deploy Branches and deploy contexts, and set Branch deploys to “All” or name your release branch. Push a candidate there, and Netlify builds it alongside main. Split Testing then lives under Site configuration Split Testing, where you pick two branches and assign percentages. The percentages sit behind a form with no CLI and no documentation in the API, so a workflow has no way to touch a test once someone has started one. So the pipeline does the split itself , in an edge function reading numbers the CLI can set. The obvious place to put those numbers is an environment variable, and it does not work. Netlify is explicit that “changes to environment variables for edge functions require a build and deploy to take effect” — each deploy captures the values as they were at deploy time. A workflow that sets CANARY PERCENT would never move the running function, and, much worse, the abort job setting it back to zero would not either. So the numbers live in Netlify Blobs https://docs.netlify.com/build/data-and-storage/netlify-blobs/ instead, which is Netlify’s runtime store: readable from an edge function, writable from CI, and no deploy in between. It is the same shape as Vercel’s Edge Config and Cloudflare’s KV. js // netlify/edge-functions/canary.js import { getStore } from '@netlify/blobs'; export default async request, context = { // Blobs default to eventual consistency, which propagates within 60 // seconds. An abort cannot wait 60 seconds, so ask for strong. const store = getStore { name: 'canary', consistency: 'strong' } ; // A missing or malformed key sends everyone to the stable version const rollout = await store.get 'rollout', { type: 'json' } ; if rollout?.percentage || rollout.origin return; // Countries stay empty for a plain percentage rollout const countries = rollout.countries ?? ; const country = context.geo?.country?.code; if countries.length && countries.includes country return; // Returning visitors keep whatever they were given const existing = context.cookies.get 'canary' ; const assigned = existing ?? Math.random 100 < rollout.percentage ? 'on' : 'off' ; context.cookies.set { name: 'canary', value: assigned, path: '/' } ; if assigned === 'off' return; return context.rewrite rollout.origin + new URL request.url .pathname ; }; export const config = { path: '/ ' }; That one file covers both the percentage rollout and the country rollout, since leaving countries empty skips the geo check. This file lives in netlify/edge-functions/ . Every command below writes that one key with netlify blobs:set , which takes effect on the next request. Deploying to five percent instead of everyone .github/workflows/deploy.yml name: Deploy on: push: branches: main concurrency: group: production-deploy cancel-in-progress: false permissions: contents: read env: NETLIFY AUTH TOKEN: ${{ secrets.NETLIFY AUTH TOKEN }} NETLIFY SITE ID: ${{ secrets.NETLIFY SITE ID }} jobs: deploy: runs-on: ubuntu-latest outputs: url: ${{ steps.deploy.outputs.url }} id: ${{ steps.deploy.outputs.id }} previous: ${{ steps.previous.outputs.id }} steps: - uses: actions/checkout@v4 - uses: ./.github/actions/setup - id: previous run: | PREV=$ netlify api listSiteDeploys --data '{"site id":"'"$NETLIFY SITE ID"'"}' \ | jq -r ' . | select .state=="ready" and .context=="production" 0 .id' echo "id=$PREV" "$GITHUB OUTPUT" Netlify deploys what you hand it, so the build happens here - run: npm run build - id: deploy run: | OUT=$ netlify deploy --dir=dist --json echo "url=$ echo "$OUT" | jq -r '.deploy url' " "$GITHUB OUTPUT" echo "id=$ echo "$OUT" | jq -r '.deploy id' " "$GITHUB OUTPUT" - name: Send it 5% of traffic run: | netlify blobs:set canary rollout \ "$ jq -nc --arg o '${{ steps.deploy.outputs.url }}' \ '{origin:$o,percentage:5}' " The smoke and soak jobs are the ones from the Vercel section, with PLAYWRIGHT BASE URL set to the draft URL and BASELINE RELEASE set to needs.deploy.outputs.previous . Taking the rest, or giving it back promote: needs: deploy, soak if: needs.soak.outputs.decision == 'promote' runs-on: ubuntu-latest steps: - uses: actions/checkout@v4 - uses: ./.github/actions/setup Publish the draft that is already up there. Rebuilding here would ship different bytes from the ones you just tested. - run: | netlify api restoreSiteDeploy --data '{ "site id": "'"$NETLIFY SITE ID"'", "deploy id": "${{ needs.deploy.outputs.id }}" }' netlify blobs:set canary rollout '{"percentage":0}' abort: needs: deploy, smoke, soak if: always && needs.smoke.result == 'failure' || needs.soak.outputs.decision == 'abort' runs-on: ubuntu-latest steps: - uses: actions/checkout@v4 - uses: ./.github/actions/setup - run: netlify blobs:set canary rollout '{"percentage":0}' - run: | curl -sS -X POST -H 'Content-type: application/json' \ --data "{\"text\": \"Canary aborted. Split closed, production unchanged.\"}" \ "${{ secrets.SLACK WEBHOOK }}" Rolling back after promotion Same two watch jobs as Vercel, with one step swapped. Deployments on Netlify are immutable, so the previous one remains available for republishing. rollback: needs: deploy, watch-15, watch-60 if: always && needs.watch-15.outputs.decision == 'abort' || needs.watch-60.outputs.decision == 'abort' runs-on: ubuntu-latest steps: - uses: actions/checkout@v4 - uses: ./.github/actions/setup - run: | netlify api restoreSiteDeploy --data '{ "site id": "'"$NETLIFY SITE ID"'", "deploy id": "${{ needs.deploy.outputs.previous }}" }' Everybody moves at once here. Your edge function returns early when CANARY PERCENT is zero, so a visitor holding a canary=on cookie stops being routed anywhere the moment the abort or promote step runs. While you investigate, “Lock to stop auto publishing” on that deploy in the dashboard stops the next merge, which would put the bad version straight back. The whole file .github/workflows/deploy.yml name: Deploy on: push: branches: main concurrency: group: production-deploy cancel-in-progress: false permissions: contents: read env: NETLIFY AUTH TOKEN: ${{ secrets.NETLIFY AUTH TOKEN }} NETLIFY SITE ID: ${{ secrets.NETLIFY SITE ID }} SENTRY TOKEN: ${{ secrets.SENTRY TOKEN }} SENTRY ORG: my-org SENTRY PROJECT ID: '4504000000000000' jobs: deploy: runs-on: ubuntu-latest outputs: url: ${{ steps.deploy.outputs.url }} id: ${{ steps.deploy.outputs.id }} baseline: ${{ steps.baseline.outputs.id }} steps: - uses: actions/checkout@v4 - uses: ./.github/actions/setup - run: npm run build - id: baseline run: | ID=$ netlify api listSiteDeploys --data '{"site id":"'"$NETLIFY SITE ID"'"}' \ | jq -r ' . | select .state=="ready" and .context=="production" 0 .id' echo "id=$ID" "$GITHUB OUTPUT" - id: deploy run: | OUT=$ netlify deploy --dir=dist --json echo "url=$ echo "$OUT" | jq -r '.deploy url' " "$GITHUB OUTPUT" echo "id=$ echo "$OUT" | jq -r '.deploy id' " "$GITHUB OUTPUT" - name: Send it 5% of traffic run: | netlify blobs:set canary rollout \ "$ jq -nc --arg o '${{ steps.deploy.outputs.url }}' \ '{origin:$o,percentage:5}' " smoke: needs: deploy runs-on: ubuntu-latest timeout-minutes: 5 steps: - uses: actions/checkout@v4 - uses: ./.github/actions/setup - run: npx playwright install --with-deps chromium - run: npx playwright test e2e/smoke --project=chromium env: PLAYWRIGHT BASE URL: ${{ needs.deploy.outputs.url }} soak: needs: deploy, smoke runs-on: ubuntu-latest environment: canary-soak outputs: decision: ${{ steps.check.outputs.decision }} steps: - uses: actions/checkout@v4 - uses: ./.github/actions/setup - id: check run: node scripts/canary-health.mjs env: CANARY RELEASE: ${{ github.sha }} BASELINE RELEASE: ${{ needs.deploy.outputs.baseline }} promote: needs: deploy, soak if: needs.soak.outputs.decision == 'promote' runs-on: ubuntu-latest steps: - uses: actions/checkout@v4 - uses: ./.github/actions/setup - run: | netlify api restoreSiteDeploy --data '{ "site id": "'"$NETLIFY SITE ID"'", "deploy id": "${{ needs.deploy.outputs.id }}" }' netlify blobs:set canary rollout '{"percentage":0}' hold: needs: soak if: needs.soak.outputs.decision == 'hold' runs-on: ubuntu-latest steps: - run: | curl -sS -X POST -H 'Content-type: application/json' \ --data "{\"text\": \"Canary parked at 5%. Not enough data to decide.\"}" \ "${{ secrets.SLACK WEBHOOK }}" abort: needs: deploy, smoke, soak if: always && needs.smoke.result == 'failure' || needs.soak.outputs.decision == 'abort' runs-on: ubuntu-latest steps: - uses: actions/checkout@v4 - uses: ./.github/actions/setup - run: netlify blobs:set canary rollout '{"percentage":0}' - run: | curl -sS -X POST -H 'Content-type: application/json' \ --data "{\"text\": \"Canary aborted. Production is back on the previous build.\"}" \ "${{ secrets.SLACK WEBHOOK }}" watch-15: needs: deploy, promote runs-on: ubuntu-latest environment: watch-15min outputs: decision: ${{ steps.check.outputs.decision }} steps: - uses: actions/checkout@v4 - uses: ./.github/actions/setup - id: check run: node scripts/canary-health.mjs env: CANARY RELEASE: ${{ github.sha }} BASELINE RELEASE: ${{ needs.deploy.outputs.baseline }} watch-60: needs: deploy, promote, watch-15 if: needs.watch-15.outputs.decision = 'abort' runs-on: ubuntu-latest environment: watch-45min runs after watch-15, so 15 + 45 = an hour in outputs: decision: ${{ steps.check.outputs.decision }} steps: - uses: actions/checkout@v4 - uses: ./.github/actions/setup - id: check run: node scripts/canary-health.mjs env: CANARY RELEASE: ${{ github.sha }} BASELINE RELEASE: ${{ needs.deploy.outputs.baseline }} rollback: needs: deploy, watch-15, watch-60 if: always && needs.watch-15.outputs.decision == 'abort' || needs.watch-60.outputs.decision == 'abort' runs-on: ubuntu-latest steps: - uses: actions/checkout@v4 - uses: ./.github/actions/setup - run: | netlify api restoreSiteDeploy --data '{ "site id": "'"$NETLIFY SITE ID"'", "deploy id": "${{ needs.deploy.outputs.baseline }}" }' - run: | curl -sS -X POST -H 'Content-type: application/json' \ --data "{\"text\": \"Rolled back after promotion.\"}" \ "${{ secrets.SLACK WEBHOOK }}" Rolling out to one country The edge function above already reads CANARY COUNTRIES , so the geo workflow is would be: start: runs-on: ubuntu-latest steps: - uses: actions/checkout@v4 - uses: ./.github/actions/setup - run: | COUNTRIES=$ echo "${{ inputs.countries }}" | jq -R 'split "," ' netlify blobs:set canary rollout \ "$ jq -nc --argjson c "$COUNTRIES" --arg o "$CANARY ORIGIN" \ '{countries:$c,percentage:10,origin:$o}' " widen: needs: soak-10 if: needs.soak-10.outputs.decision == 'promote' runs-on: ubuntu-latest steps: - uses: actions/checkout@v4 - uses: ./.github/actions/setup Widen without touching the other keys - run: | CURRENT=$ netlify blobs:get canary rollout netlify blobs:set canary rollout \ "$ jq -c '.percentage = 50' <<<"$CURRENT" " stop: needs: soak-10 if: always && needs.soak-10.outputs.decision == 'abort' runs-on: ubuntu-latest steps: - uses: actions/checkout@v4 - uses: ./.github/actions/setup - run: netlify blobs:set canary rollout '{"percentage":0}' Cloudflare In the words of Tejas Kumar: “I goon to Cloudflare” Cloudflare splits traffic between Worker versions, and it keeps uploading a version separate from deploying one, which makes it pleasant to automate. WRANGLER OUTPUT FILE PATH=out.ndjson npx wrangler versions upload That returns an ID and a preview URL, and the version carries zero production traffic until you say otherwise. Your smoke tests then run against the preview before any split exists. To have it working properly you need a header Cloudflare-Workers-Version-Key , so that every request with the same value goes to the same version. In the dashboard, open your zone, then Rules, then Transform Rules, then Modify Request Header. When incoming requests match: http.host eq "app.example.com" Then: Set dynamic → Header name: Cloudflare-Workers-Version-Key Value: http.request.cookies "sid" 0 Swap sid for your session cookie and set that cookie in the first response, since visitors arriving without one fall back to per-request splitting. The same rule as Terraform, if your zone lives in code: resource "cloudflare ruleset" "version affinity" { zone id = var.zone id kind = "zone" phase = "http request late transform" rules { action = "rewrite" expression = " http.host eq \"app.example.com\" " action parameters { headers { name = "Cloudflare-Workers-Version-Key" operation = "set" expression = "http.request.cookies \"sid\" 0 " } } } } One more binding, so your metrics have a version to group by. The runtime hands you the ID, which means Cloudflare skips the build-time Sentry release step from earlier. // wrangler.jsonc { "version metadata": { "binding": "CF VERSION METADATA" } } // run npx wrangler types after adding the binding so env is typed export default { async fetch request: Request, env: Env { const { id: versionId } = env.CF VERSION METADATA; // attach versionId to your Sentry scope and your analytics events }, }; Deploying to five percent instead of everyone .github/workflows/deploy.yml name: Deploy on: push: branches: main concurrency: group: production-deploy cancel-in-progress: false permissions: contents: read env: CLOUDFLARE API TOKEN: ${{ secrets.CLOUDFLARE API TOKEN }} CLOUDFLARE ACCOUNT ID: ${{ secrets.CLOUDFLARE ACCOUNT ID }} jobs: deploy: runs-on: ubuntu-latest outputs: version: ${{ steps.upload.outputs.id }} preview: ${{ steps.upload.outputs.preview }} previous: ${{ steps.previous.outputs.id }} steps: - uses: actions/checkout@v4 - uses: ./.github/actions/setup - run: npm run build - id: previous run: | PREV=$ npx wrangler deployments list --json \ | jq -r '. 0 .versions 0 .version id' echo "id=$PREV" "$GITHUB OUTPUT" Run this once locally and read the JSON before trusting either path. An empty preview URL sends Playwright to localhost and passes for the wrong reason. - id: upload env: WRANGLER OUTPUT FILE PATH: ${{ runner.temp }}/wrangler.ndjson run: | npx wrangler versions upload ND-JSON, one object per line. The version-upload entry has both. ENTRY=$ jq -sc ' . | select .type=="version-upload" | last' \ "$WRANGLER OUTPUT FILE PATH" echo "id=$ jq -r '.version id' <<<"$ENTRY" " "$GITHUB OUTPUT" echo "preview=$ jq -r '.preview url' <<<"$ENTRY" " "$GITHUB OUTPUT" - name: Send it 5% of traffic run: | npx wrangler versions deploy \ "${{ steps.upload.outputs.id }}@5%" \ "${{ steps.previous.outputs.id }}@95%" --yes Point PLAYWRIGHT BASE URL in the smoke job at needs.deploy.outputs.preview , and set CANARY RELEASE in the soak job to needs.deploy.outputs.version . Taking the rest, or giving it back promote: needs: deploy, soak if: needs.soak.outputs.decision == 'promote' runs-on: ubuntu-latest steps: - uses: actions/checkout@v4 - uses: ./.github/actions/setup A single version at 100% ends the split - run: npx wrangler versions deploy "${{ needs.deploy.outputs.version }}@100%" --yes abort: needs: deploy, smoke, soak if: always && needs.smoke.result == 'failure' || needs.soak.outputs.decision == 'abort' runs-on: ubuntu-latest steps: - uses: actions/checkout@v4 - uses: ./.github/actions/setup - run: npx wrangler versions deploy "${{ needs.deploy.outputs.previous }}@100%" --yes Rolling back after promotion rollback: needs: deploy, watch-15, watch-60 if: always && needs.watch-15.outputs.decision == 'abort' || needs.watch-60.outputs.decision == 'abort' runs-on: ubuntu-latest steps: - uses: actions/checkout@v4 - uses: ./.github/actions/setup - run: npx wrangler rollback --message "post-promotion health check" --yes It reaches the edge in seconds, and a split deployment collapses onto the version you picked. Version affinity stops applying at that point, since one version is now serving everything, so the sessions you pinned move along with everybody else. The whole file .github/workflows/deploy.yml name: Deploy on: push: branches: main concurrency: group: production-deploy cancel-in-progress: false permissions: contents: read env: CLOUDFLARE API TOKEN: ${{ secrets.CLOUDFLARE API TOKEN }} CLOUDFLARE ACCOUNT ID: ${{ secrets.CLOUDFLARE ACCOUNT ID }} SENTRY TOKEN: ${{ secrets.SENTRY TOKEN }} SENTRY ORG: my-org SENTRY PROJECT ID: '4504000000000000' jobs: deploy: runs-on: ubuntu-latest outputs: url: ${{ steps.upload.outputs.preview }} version: ${{ steps.upload.outputs.id }} baseline: ${{ steps.baseline.outputs.id }} steps: - uses: actions/checkout@v4 - uses: ./.github/actions/setup - run: npm run build - id: baseline run: | ID=$ npx wrangler deployments list --json \ | jq -r '. 0 .versions 0 .version id' echo "id=$ID" "$GITHUB OUTPUT" - id: upload env: WRANGLER OUTPUT FILE PATH: ${{ runner.temp }}/wrangler.ndjson run: | npx wrangler versions upload ND-JSON, one object per line. The version-upload entry has both. ENTRY=$ jq -sc ' . | select .type=="version-upload" | last' \ "$WRANGLER OUTPUT FILE PATH" echo "id=$ jq -r '.version id' <<<"$ENTRY" " "$GITHUB OUTPUT" echo "preview=$ jq -r '.preview url' <<<"$ENTRY" " "$GITHUB OUTPUT" - name: Send it 5% of traffic run: | npx wrangler versions deploy \ "${{ steps.upload.outputs.id }}@5%" \ "${{ steps.baseline.outputs.id }}@95%" --yes smoke: needs: deploy runs-on: ubuntu-latest timeout-minutes: 5 steps: - uses: actions/checkout@v4 - uses: ./.github/actions/setup - run: npx playwright install --with-deps chromium - run: npx playwright test e2e/smoke --project=chromium env: PLAYWRIGHT BASE URL: ${{ needs.deploy.outputs.url }} soak: needs: deploy, smoke runs-on: ubuntu-latest environment: canary-soak outputs: decision: ${{ steps.check.outputs.decision }} steps: - uses: actions/checkout@v4 - uses: ./.github/actions/setup - id: check run: node scripts/canary-health.mjs env: CANARY RELEASE: ${{ needs.deploy.outputs.version }} BASELINE RELEASE: ${{ needs.deploy.outputs.baseline }} promote: needs: deploy, soak if: needs.soak.outputs.decision == 'promote' runs-on: ubuntu-latest steps: - uses: actions/checkout@v4 - uses: ./.github/actions/setup - run: npx wrangler versions deploy "${{ needs.deploy.outputs.version }}@100%" --yes hold: needs: soak if: needs.soak.outputs.decision == 'hold' runs-on: ubuntu-latest steps: - run: | curl -sS -X POST -H 'Content-type: application/json' \ --data "{\"text\": \"Canary parked at 5%. Not enough data to decide.\"}" \ "${{ secrets.SLACK WEBHOOK }}" abort: needs: deploy, smoke, soak if: always && needs.smoke.result == 'failure' || needs.soak.outputs.decision == 'abort' runs-on: ubuntu-latest steps: - uses: actions/checkout@v4 - uses: ./.github/actions/setup - run: npx wrangler versions deploy "${{ needs.deploy.outputs.baseline }}@100%" --yes - run: | curl -sS -X POST -H 'Content-type: application/json' \ --data "{\"text\": \"Canary aborted. Production is back on the previous build.\"}" \ "${{ secrets.SLACK WEBHOOK }}" watch-15: needs: deploy, promote runs-on: ubuntu-latest environment: watch-15min outputs: decision: ${{ steps.check.outputs.decision }} steps: - uses: actions/checkout@v4 - uses: ./.github/actions/setup - id: check run: node scripts/canary-health.mjs env: CANARY RELEASE: ${{ needs.deploy.outputs.version }} BASELINE RELEASE: ${{ needs.deploy.outputs.baseline }} watch-60: needs: deploy, promote, watch-15 if: needs.watch-15.outputs.decision = 'abort' runs-on: ubuntu-latest environment: watch-45min runs after watch-15, so 15 + 45 = an hour in outputs: decision: ${{ steps.check.outputs.decision }} steps: - uses: actions/checkout@v4 - uses: ./.github/actions/setup - id: check run: node scripts/canary-health.mjs env: CANARY RELEASE: ${{ needs.deploy.outputs.version }} BASELINE RELEASE: ${{ needs.deploy.outputs.baseline }} rollback: needs: deploy, watch-15, watch-60 if: always && needs.watch-15.outputs.decision == 'abort' || needs.watch-60.outputs.decision == 'abort' runs-on: ubuntu-latest steps: - uses: actions/checkout@v4 - uses: ./.github/actions/setup - run: npx wrangler rollback --message "post-promotion health check" --yes - run: | curl -sS -X POST -H 'Content-type: application/json' \ --data "{\"text\": \"Rolled back after promotion.\"}" \ "${{ secrets.SLACK WEBHOOK }}" Rolling out to one country The Worker reads its numbers from a KV namespace, so create one and bind it first. npx wrangler kv namespace create CONFIG // wrangler.jsonc { "kv namespaces": { "binding": "CONFIG", "id": "0f2ac74b498b48028cb68387c421e279" } } export default { async fetch request, env { // cacheTtl keeps this off the hot path; 60s is fast enough to abort const rollout = await env.CONFIG.get 'canary', { type: 'json', cacheTtl: 60, } ; // A missing or malformed key sends everyone to the stable version if rollout?.countries?.length return fetch request ; const country = request.cf?.country; if country || rollout.countries.includes country { return fetch request ; } const cookie = request.headers.get 'cookie' ?? ''; const existing = cookie.match /canary= on|off / ?. 1 ; const assigned = existing ?? Math.random 100 < rollout.percentage ? 'on' : 'off' ; let upstream; if assigned === 'on' { const url = new URL request.url ; url.hostname = rollout.canaryHost; upstream = await fetch new Request url, request ; } else { upstream = await fetch request ; } const response = new Response upstream.body, upstream ; response.headers.append 'set-cookie', canary=${assigned}; Path=/; Max-Age=86400; SameSite=Lax ; return response; }, }; Keep the guard on a missing key. A Worker that throws when somebody deletes that KV entry takes the whole site down with it. - run: | COUNTRIES=$ echo "${{ inputs.countries }}" | jq -R 'split "," ' npx wrangler kv key put --binding CONFIG --remote canary \ "$ jq -nc --argjson c "$COUNTRIES" \ '{countries:$c,percentage:10,canaryHost:"canary.example.com"}' " Deploying it yourself on GCP or AWS Putting the app in a container means nobody hands you a traffic splitter anymore, so you configure the one your platform already has. Cloud Run A Cloud Run service keeps every revision you have ever deployed, and each deployment is split across them. Deploying with --no-traffic --tag canary ships a revision that serves nobody and gets its own URL, which is the same preview step Cloudflare gives you. gcloud run deploy my-app \ --image gcr.io/my-project/my-app:$SHA \ --region us-central1 --no-traffic --tag canary reachable at https://canary---my-app-xxxxx.a.run.app Traffic then moves by tag. gcloud run services update-traffic my-app --region us-central1 --to-tags canary=5 gcloud run services update-traffic my-app --region us-central1 --to-tags canary=50 gcloud run services update-traffic my-app --region us-central1 --to-tags canary=100 Rolling back names a revision instead of a tag, which is why the workflow captures the current one before deploying anything. gcloud run services update-traffic my-app --region us-central1 \ --to-revisions my-app-00042-abc=100 gcloud run services update my-app --region us-central1 --session-affinity Use Workload Identity Federation for the credentials. It lets the workflow assume a service account with no JSON key sitting in GitHub, which is why every job below asks for id-token: write . .github/workflows/deploy.yml name: Deploy on: push: branches: main concurrency: group: production-deploy cancel-in-progress: false env: SERVICE: my-app REGION: us-central1 SENTRY TOKEN: ${{ secrets.SENTRY TOKEN }} SENTRY ORG: my-org SENTRY PROJECT ID: '4504000000000000' jobs: deploy: runs-on: ubuntu-latest permissions: contents: read id-token: write Workload Identity Federation, no JSON keys outputs: preview: ${{ steps.deploy.outputs.preview }} revision: ${{ steps.deploy.outputs.revision }} baseline: ${{ steps.baseline.outputs.revision }} steps: - uses: actions/checkout@v4 - uses: google-github-actions/auth@v2 with: workload identity provider: ${{ secrets.GCP WIF PROVIDER }} service account: ${{ secrets.GCP SERVICE ACCOUNT }} - uses: google-github-actions/setup-gcloud@v2 The entry carrying the most traffic, rather than whichever one happens to sit first in the list - id: baseline run: | REV=$ gcloud run services describe "$SERVICE" --region="$REGION" --format=json \ | jq -r ' .status.traffic | select .percent 0 | max by .percent | .revisionName' echo "revision=$REV" "$GITHUB OUTPUT" - run: | gcloud builds submit --tag "gcr.io/$GCP PROJECT/$SERVICE:${{ github.sha }}" env: GCP PROJECT: ${{ secrets.GCP PROJECT }} --no-traffic ships it with zero users. --tag gives it a URL. - id: deploy run: | gcloud run deploy "$SERVICE" \ --image "gcr.io/${{ secrets.GCP PROJECT }}/$SERVICE:${{ github.sha }}" \ --region "$REGION" --no-traffic --tag canary URL=$ gcloud run services describe "$SERVICE" --region="$REGION" --format=json \ | jq -r '.status.traffic | select .tag=="canary" | .url' REV=$ gcloud run services describe "$SERVICE" --region="$REGION" \ --format='value status.latestCreatedRevisionName ' echo "preview=$URL" "$GITHUB OUTPUT" echo "revision=$REV" "$GITHUB OUTPUT" - name: Send it 5% of traffic run: gcloud run services update-traffic "$SERVICE" --region="$REGION" --to-tags canary=5 smoke: needs: deploy runs-on: ubuntu-latest timeout-minutes: 5 steps: - uses: actions/checkout@v4 - uses: ./.github/actions/setup - run: npx playwright install --with-deps chromium - run: npx playwright test e2e/smoke --project=chromium env: PLAYWRIGHT BASE URL: ${{ needs.deploy.outputs.preview }} soak: needs: deploy, smoke runs-on: ubuntu-latest environment: canary-soak outputs: decision: ${{ steps.check.outputs.decision }} steps: - uses: actions/checkout@v4 - uses: ./.github/actions/setup - id: check run: node scripts/canary-health.mjs env: CANARY RELEASE: ${{ github.sha }} BASELINE RELEASE: ${{ needs.deploy.outputs.baseline }} promote: needs: deploy, soak if: needs.soak.outputs.decision == 'promote' runs-on: ubuntu-latest permissions: contents: read id-token: write steps: - uses: google-github-actions/auth@v2 with: workload identity provider: ${{ secrets.GCP WIF PROVIDER }} service account: ${{ secrets.GCP SERVICE ACCOUNT }} - uses: google-github-actions/setup-gcloud@v2 - run: gcloud run services update-traffic "$SERVICE" --region="$REGION" --to-tags canary=100 hold: needs: soak if: needs.soak.outputs.decision == 'hold' runs-on: ubuntu-latest steps: - run: | curl -sS -X POST -H 'Content-type: application/json' \ --data "{\"text\": \"Canary parked at 5%. Not enough data to decide.\"}" \ "${{ secrets.SLACK WEBHOOK }}" abort: needs: deploy, smoke, soak if: always && needs.smoke.result == 'failure' || needs.soak.outputs.decision == 'abort' runs-on: ubuntu-latest permissions: contents: read id-token: write steps: - uses: google-github-actions/auth@v2 with: workload identity provider: ${{ secrets.GCP WIF PROVIDER }} service account: ${{ secrets.GCP SERVICE ACCOUNT }} - uses: google-github-actions/setup-gcloud@v2 - run: | gcloud run services update-traffic "$SERVICE" --region="$REGION" \ --to-revisions="${{ needs.deploy.outputs.baseline }}=100" - run: | curl -sS -X POST -H 'Content-type: application/json' \ --data "{\"text\": \"Canary aborted. All traffic back on the previous revision.\"}" \ "${{ secrets.SLACK WEBHOOK }}" watch-15: needs: deploy, promote runs-on: ubuntu-latest environment: watch-15min outputs: decision: ${{ steps.check.outputs.decision }} steps: - uses: actions/checkout@v4 - uses: ./.github/actions/setup - id: check run: node scripts/canary-health.mjs env: CANARY RELEASE: ${{ github.sha }} BASELINE RELEASE: ${{ needs.deploy.outputs.baseline }} watch-60: needs: deploy, promote, watch-15 if: needs.watch-15.outputs.decision = 'abort' runs-on: ubuntu-latest environment: watch-45min runs after watch-15, so 15 + 45 = an hour in outputs: decision: ${{ steps.check.outputs.decision }} steps: - uses: actions/checkout@v4 - uses: ./.github/actions/setup - id: check run: node scripts/canary-health.mjs env: CANARY RELEASE: ${{ github.sha }} BASELINE RELEASE: ${{ needs.deploy.outputs.baseline }} rollback: needs: deploy, watch-15, watch-60 if: always && needs.watch-15.outputs.decision == 'abort' || needs.watch-60.outputs.decision == 'abort' runs-on: ubuntu-latest permissions: contents: read id-token: write steps: - uses: google-github-actions/auth@v2 with: workload identity provider: ${{ secrets.GCP WIF PROVIDER }} service account: ${{ secrets.GCP SERVICE ACCOUNT }} - uses: google-github-actions/setup-gcloud@v2 - run: | gcloud run services update-traffic "$SERVICE" --region="$REGION" \ --to-revisions="${{ needs.deploy.outputs.baseline }}=100" ECS behind an Application Load Balancer In ECS, containers use the load balancer split, where a listener rule forwards traffic to two target groups with weights. aws elbv2 modify-listener --listener-arn "$LISTENER" --default-actions ' { "Type": "forward", "ForwardConfig": { "TargetGroups": {"TargetGroupArn": "'"$BLUE"'", "Weight": 95}, {"TargetGroupArn": "'"$GREEN"'", "Weight": 5} , "TargetGroupStickinessConfig": {"Enabled": true, "DurationSeconds": 3600} } } ' Deploy registers a task definition and updates the green service. The ramp is that modify-listener call with new weights, promoting set green to 100, and both abort and rollback set blue back to 100. - name: Send it 5% of traffic run: | aws ecs update-service --cluster "$CLUSTER" --service "$SERVICE GREEN" \ --task-definition "${{ steps.taskdef.outputs.arn }}" --force-new-deployment aws ecs wait services-stable --cluster "$CLUSTER" --services "$SERVICE GREEN" ./scripts/set-weights.sh 95 5 promote - run: ./scripts/set-weights.sh 0 100 abort and rollback are the same call, the other way round - run: ./scripts/set-weights.sh 100 0 bash /usr/bin/env bash scripts/set-weights.sh