Part 1 was Jenkins as code with ephemeral workers. Part 2 was macOS. This one moves a chunk of the CI workload off Jenkins entirely - onto GitHub Actions, with EC2 spot as the runner fleet. Jenkins isn't dead here. It still handles the big builds - macOS, Windows, anything that runs for hours or needs custom orchestration. GitHub Actions runs alongside it for the workloads where it fits better. This post covers the self-hosted spot runner pattern: how to point GitHub Actions at your own ephemeral EC2 fleet, and what bites once you do. GitHub's managed runners are fine for small teams. A few things push you toward self-hosted:
- Cost at volume. GitHub bills its managed Linux runners at $0.008/minute ($0.48/hour). Fine for a few builds a day. Last month we ran 80,887 runner-minutes across 29,347 jobs (~1,350 hours). On managed runners that would have been ~$647. Our actual EC2 bill for the runner fleet was ~$160 - $130 on spot, $28 EC2-Other (EBS, ENIs, data transfer). Roughly 4x cheaper, and the gap widens the more you run.
- Instance shape. Managed runners come in fixed sizes. If your build wants 16 vCPUs and 64 GB of RAM, or a GPU, or arm64, you're either paying for the largest tier or stuck. Self-hosted lets you pick the EC2 type the build needs.
- Network access. Builds that talk to private resources - internal artifact registries, RDS, anything behind a VPC - are awkward on managed runners. Self-hosted runners live inside your VPC, no proxies or tunnels. The bill got us to try it. The VPC access and instance flexibility were bonuses. A self-hosted GitHub Actions runner is a small agent that registers with a GitHub repo or org, polls for jobs matching its labels, runs them, reports back. It doesn't care where it lives - bare metal, VM, container, anything that runs the binary. Two flavors: We picked ephemeral. A long-lived self-hosted runner means babysitting a host, build pollution from a shared agent, and a security blast radius that never closes. So every job gets its own EC2 spot instance from a Packer-baked AMI. Job runs, instance terminates. Same one-build-per-worker pattern as the Jenkins fleet, different control plane. There's no single "self-hosted spot runner" service. You stitch it yourself, or grab one of the open-source modules that stitch it for you. I went with terraform-aws-github-runner - most battle-tested of the bunch, drops into a Terraform-managed AWS account without much ceremony. (If you remember it as philips-labs/terraform-aws-github-runner
- same project, moved to the github-aws-runners org.) When someone opens a PR that triggers a workflow: workflow_job webhook the moment a job is queued.--ephemeral , so the agent exits after one job. A scale-down Lambda comes by on a schedule and reaps anything left over.The Lambdas, SQS queues, IAM - all of it lives inside the module. You write the Terraform that declares which runners exist, which AMI they use, which instance types are eligible. 📌 IMAGE TODO (architecture): linear flow left-to-right - GitHub webhook → API Gateway + Lambda → SQS → Scale-up Lambda → EC2 spot instance (Packer AMI) → Job runs → Instance terminates. One dashed loop labeled "scale-down Lambda watches and cleans up". Hand-drawn excalidraw style to match Parts 1 and 2. The setup earns its keep once you split runners into tiers by label. Workflows pick a tier with runs-on: in their YAML. We run three: t3.medium / m5.large . Linters, formatters, doc builds. Cheap, spot capacity is never a problem, spawns fast.m5.xlarge / c5.xlarge . The typical build/test workflow.c7a.4xlarge / c8a.8xlarge . Compile-heavy builds, big test suites, anything that scales with cores.Each tier is its own call to the same Terraform module with different labels and instance-type lists. Sanitized:
module "github-runners" {
source = "github-aws-runners/github-runner/aws//modules/multi-runner"
version = "~> 6.0"
multi_runner_config = {
"linux-x64-small" = {
runner_config = {
runner_extra_labels = "linux,x64,small" instance_types = local.default_instances ami_filter = { name = ["ci-runner-x64"] } enable_ephemeral_runners = true enable_spot_instances = true
}
}
"linux-x64-compute-intensive" = {
runner_config = {
runner_extra_labels = "linux,x64,compute-intensive" instance_types = local.compute_intensive ami_filter = { name = ["ci-runner-x64"] } enable_ephemeral_runners = true enable_spot_instances = true
}
}
}
github_app = { ... }
webhook_secret = random_id.webhook_secret.hex vpc_id = module.vpc.vpc_id subnet_ids = module.vpc.private_subnets } Picking the tier in a workflow: jobs: build: runs-on: [self-hosted, linux, x64, compute-intensive] steps:
- uses: actions/checkout@v4
- run: make build
GitHub matches the runs-on array against runner labels and picks any registered runner that has them all. The Lambda only spawns instances on demand, so an unused tier costs nothing. Pure on-demand scaling looks great on paper: zero idle runners, pay only when a job runs, Lambda spawns exactly when GitHub queues something. Two things spoil it. The morning-rush problem. The first PRs of the day all queue around the time people log in. On pure on-demand, every one eats the full cold-start (~60-120s from queue to running). A dozen devs hitting "push" at 9am and you have a backlog people notice. The 3am problem. Even on spot, idle runners cost something - EBS volumes on the warm AMIs, plus the always-on orchestration Lambdas. Outside business hours the queue is mostly empty. No point keeping capacity hot. The module handles both with idle pools and scheduled scaling. What we do:
The Terraform for it (sanitized, per-tier):
runner_config = {
enable_ephemeral_runners = true enable_spot_instances = true
idle_config = [
{
cron = "0 8 * * MON-FRI" # ramp up at 08:00 weekdays
timeZone = "Europe/Berlin" idleCount = 3
},
{
cron = "0 20 * * MON-FRI" # ramp down at 20:00 weekdays
timeZone = "Europe/Berlin" idleCount = 0
},
]
}
A few notes: During working hours the dev experience is about as snappy as managed runners. Outside, the bill is close to zero. 📌 IMAGE TODO (warm-pool schedule): a 24-hour timeline showing pool size. Flat at 0 from 00:00-08:00, ramps up to N runners at 08:00, stays at N until 20:00, drops to 0 for the night. Weekend bar shows pool=0 all day. Annotate "cold-start hides here" inside the 08:00-20:00 band. Same idea as Part 1: whatever the runner needs - language runtimes, build tools, Docker, cached dependencies - goes into a Packer AMI ahead of time. The AMI is versioned, lives in your AWS account, and is referenced by the runner's ami_filter . For GitHub Actions the image is usually lighter than the Jenkins worker AMIs - GHA workflows install most of their tooling at runtime via setup-node , setup-python , setup-java . So the base image just needs: docker build / docker run ).git , curl , jq , unzip .The image ends up 5-10 GB. A fresh spot instance pulls it and starts running in well under a minute. First question everyone asks about spot for CI: what happens when AWS yanks the instance mid-build? The build fails. GitHub re-queues. A fresh instance picks it up. No recovery dance to write - the runner is ephemeral and the workflow should be idempotent anyway. You lose one partial build, the retry runs from scratch. Spot gives you 2 minutes of warning before termination. The runner listens for that and de-registers from GitHub cleanly before shutdown - the module wires this up for you. Without it, GitHub briefly shows a "runner went offline mid-job" error before the retry. Ugly but not fatal. In practice we see 1-3% interruption on the default and large tiers, a bit higher on compute-intensive (larger instance types have less spot capacity per AZ). For most workloads that's a fine trade. For workflows that can't tolerate a retry - release builds, deploys with side effects - I either flip enable_spot_instances = false for that tier (same module) or ship them over to Jenkins, where the lifecycle is more controlled. "Should this run on Jenkins or GitHub Actions?" comes up a lot. How I think about it: The two systems don't overlap, they cover different shapes of work. Moving everything to GitHub Actions would have been a mistake. Moving the small PR-scoped stuff off Jenkins freed up real capacity for the big jobs. Same pattern across all three parts: ephemeral workers, baked images, orchestrated from git, secrets from a vault. Jenkins is one way to wire it up. GitHub Actions on self-hosted spot is another. You don't have to pick one. The part you can't get wrong is the worker lifecycle - don't keep workers between builds. After that, the rest (Jenkins or GHA, spot or on-demand, Tart or vSphere) is swappable. That's where the Odyssey wraps for now. Three posts. Same advice underneath: stop clicking, bake your images, don't reuse workers. If it saves you a week, worth it. philips-labs/terraform-aws-github-runner .)Part 3 of My CI/CD Odyssey. Thanks for reading. If you run self-hosted CI differently, I'd be curious to hear about it in the comments.