{"slug": "fast-counterfactuals-for-go-cache-hit-rates", "title": "Fast Counterfactuals for Go Cache Hit Rates", "summary": "CloudX reported that 86% of actions/setup-go test package runs are pre-empted by its improved GitHub Actions caching strategy, according to the company's post on scaling Go CI. Because the company never benchmarked cloudx-io/setup-go against actions/setup-go simultaneously, it backtested the claim over a 4,000-commit range using go list build IDs to detect test package changes without running tests. CloudX said replaying or simulating 4,000 commits was infeasible because running the suite at an optimistic two minutes per commit would take more than five days per tested treatment.", "body_md": "The *pièce de résistance* in [‘Scaling Golang CI by\nReplacing `actions/setup-go`’](https://www.cloudx.ai/posts/setup-go) is the conclusion that 86%\nof `actions/setup-go` test package runs are pre-empted by our\nimproved caching strategy. To summarize the CloudX article, the puzzle\nis to optimize use of a fine-grain cache we don’t control (the\n`GOCACHE`) which is saved and loaded through a coarse-grain\ncache where we can control *cache keys* that determine which\ninstances of the `GOCACHE` are loaded and saved. CloudX makes\nthat outer, coarse-grained GitHub Actions cache more granular by writing\nto it often and restoring fresher entries.\n\nThis comparison between our key strategy and GitHub’s is a counterfactual comparison: the course of action we took against a hypothetical alternative. These are tricky!\n\nWe didn’t foresee open-sourcing `cloudx-io/setup-go`, so\nwe never ran it alongside `actions/setup-go` simultaneously\nto benchmark their relative performance; we just switched, realized\nsavings, and never looked back.\n\nTo claim a general improvement — not just an improvement for a short, potentially unrepresentative period — I needed to show an advantage over an extended interval, including different paces of development on different kinds of features. I picked a 4,000-commit range, our latest few months of development.\n\nHow would you backtest GitHub Actions performance for 4,000 commits?\nMaybe *replay* comes to mind. We have the full commit history; we\ncould create a repository using each Actions strategy and apply each\nmonorepo commit one at a time, recording cache-hit rates at each commit.\nAlternatively, we could *simulate* the Actions behavior locally\nand using the `GOCACHE` environment variable to point\n`go test` runs at the faux-Actions cache to “restore.”\n\nNeither of these approaches works because running tests is slow. Even\nif the suite takes two minutes to run per commit on average\n(optimistic), testing each of 4,000 commits in series would take more\nthan five days per tested treatment.<sup>1</sup>\n\nFor the moment, let’s set aside GitHub and focus on the fine-grained\nrecord of test runs in `GOCACHE`. What causes a test cache\nmiss? When you change a test package from whatever version populated the\ncache, the next `go test` will miss the cache and every test\ncase will run anew. When the tests run, the command output notes how\nlong they took; when they don’t run because the cache held a prior\nresult, the output notes that instead:\n\n``` bash\n$ go test ./...\nok      lukasschwab.me/demo/pkg/place   0.319s\nok      lukasschwab.me/demo/pkg/time    (cached)\n```\n\nBecause a change that requires a fresh test run could occur in the\ntest package itself *or* in any of its transitive dependencies,\npredicting test reruns isn’t as simple as checking whether the source\nfiles were updated. Thankfully, we can detect package changes without\nrunning `go test` *or* modeling package dependencies\nourselves! The `go list` tool emits a *build ID* that\nuniquely represents the test package logic, transitive dependencies\nincluded:\n\n```\ngo list -export -json -test ./... > list-output.json\n```\n\nI recommend poking around that output if you’re curious how the Go\nbuild tools process the code you feed them. For our purposes, we just\nneed to know the build IDs for test packages, which have import paths\nending in `.test`:\n\n``` bash\n$ jq -cr 'select(.ImportPath | endswith(\".test\")) | .BuildID' list-output.json\nIbwMxcbMdIQEtYuf-iSx/4BSEJASXWdJsxiNf26cO\nwIuIuummvmYTa4K267jr/weIdkEXe9zK5LWUoPeyV\n…\n```\n\nThis command builds the test binaries and exactly identifies them\nwith build IDs, but never actually runs any tests. When the build ID for\na given test package changes, the cached output no longer attests to the\nvalidity of the current build. `go test` misses the cache and\nruns the full test package from scratch.\n\n``` bash\n# Get some baseline build IDs for a demo module.\n$ go list -export -json -test ./... \\\n    | jq -cr 'select(.ImportPath | endswith(\".test\")) | {(.ImportPath): .BuildID}'\n{\"lukasschwab.me/pkg/place.test\":\"IbwMxcbMdIQEtYuf-iSx/4BSEJASXWdJsxiNf26cO\"}\n{\"lukasschwab.me/pkg/time.test\":\"wIuIuummvmYTa4K267jr/weIdkEXe9zK5LWUoPeyV\"}\n\n# Run tests to populate the GOCACHE.\n$ go test ./...\nok      lukasschwab.me/pkg/place   0.319s\nok      lukasschwab.me/pkg/time    0.313s\n\n# Modify one test package.\n$ echo \"const A = iota\" >> pkg/time/time_test.go\n\n# The build ID for time.test is changed...\n$ go list -export -json -test ./... \\\n    | jq -cr 'select(.ImportPath | endswith(\".test\")) | {(.ImportPath): .BuildID}'\n{\"lukasschwab.me/pkg/place.test\":\"IbwMxcbMdIQEtYuf-iSx/4BSEJASXWdJsxiNf26cO\"}\n{\"lukasschwab.me/pkg/time.test\":\"12RAxSa7r2gXeNxG9oHO/HDMEw48BALCftV8lUnlj\"}\n\n# ...so it gets a fresh test run!\n$ go test ./...\nok      lukasschwab.me/pkg/place   (cached)\nok      lukasschwab.me/pkg/time    0.386s\n```\n\nIn short, we can use `go list` outputs to take one commit\nand understand what test results it’ll store in the\n`GOCACHE`; then we can use `go list` to see, for\nanother commit, which test packages can be skipped on the basis of those\nstored results and which test packages, changed between the two commits,\nneed fresh runs.\n\nRealizing this fast approach for counting test package runs without\nrunning tests was the crux of the CloudX backtest: I generated test\nbuild ID sets for each of the 4000 commits.[<sup>2</sup>](#fn2) By\nmodeling the `actions/setup-go` and\n`cloudx-io/setup-go` key-lookups in the coarse-grained GitHub\nActions cache, I identified which pairs of commits would write and\nrestore that cache, then I used build IDs to estimate the 86%\nimprovement.\n\nWe can assess further changes to the cache key scheme by backtesting it against the same build ID dataset — these are just more counterfactuals.\n\nClose-readers of [‘Poisoning a Go\nCache’](./poisoning-go-cache.html) might be surprised by my focus on *build IDs* here.\nBuild IDs don’t appear in the Go cache, it’s true! The cache stores test\noutcomes under a test *action ID.* The built test packages are\none factor in the action ID hash, but a change in any factor can — if\nwe’re being precise — cause a cache miss, and therefore a test\nrerun.\n\nIn practice, this means our backtest underestimates the number of\ntest runs for both the control and treatment key schemes. The absolute\nmeasurement effect should be roughly the same for both branches, but\nunderestimation *slightly* skews percentage-improvement\nestimates. Sue me!\n\nThis backtest also ignores timing effects: it assumes the GitHub\nActions cache includes *all prior commits’ outputs* before the\nnext commit’s `setup-go` step picks one of those outputs to\nload. Depending on how you configure your repository, commits may merge\nin quick succession and their CI may run in parallel, using staler\ncaches than our idealized model suggests.\n\nIt’s harder to say how this affects our measurements. Loading a\nstaler cache typically prolongs a CI job, predisposing the next job in\nthe sequence to also load a staler cache. The two `setup-go`\nstrategies also affect job runtimes through factors other than\ntest-cache hits, because they save and load caches of different sizes\n(see discussion of ‘pruning’ in the CloudX post).\n\nNevertheless, I’m pretty satisfied with the experimental method here, especially because it should allow another team to make an adoption decision tailored to their codebase: different Go dependency trees, subject to different development patterns, will exhibit different cache invalidation patterns even with an ideal cache.\n\n`cloudx-io/setup-go` works for us! Your mileage may vary,\nand now you can measure by how much.\n\nInstead, one could sample commits from the range and extrapolate from there.\n\n`setup-go` cache keys for each commit in the\nrange (cheap).`GOCACHE=/tmp/cacheB go test /b/...` for `GOCACHE=/tmp/cacheC go test /c/...` for `GOCACHE=/tmp/cacheB go test /a/...` against\n`GOCACHE=/tmp/cacheC go test /a/...`.\nYou could test samples in parallel, but those cold builds in step 3\nare expensive. There may be some mark-and-sweep approach for isolating\n`cacheB` and `cacheC` contents from a shared\ncache, but at some point these optimizations introduce measurement risk.\nI didn’t bother.[↩︎](#fnref1)\n\nOriginally I wanted a script I could run with\n`git rebase --exec` (great trick for running code against\nevery commit in a range), but our real repo history proved too\ncomplicated. Occasionally `main` includes a failing build,\nfor example. Also, I wanted parallelism.\n\nMy bodge looped over commits in a range and fed them to a workerpool.\nEach worker managed a worktree, checked out a commit, ran\n`go list -trimpath` (the `-trimpath` term is\nnecessary for comparing build IDs between worktrees), and recorded the\nresults. Every several commits, the script cleared and re-warmed the\nshared `GOCACHE` to prevent it exhausting available\nstorage.\n\nMy work laptop was essentially unusable the whole time.[↩︎](#fnref2)", "url": "https://wpnews.pro/news/fast-counterfactuals-for-go-cache-hit-rates", "canonical_source": "https://lukasschwab.me/blog/gen/fast-counterfactuals-for-go-cache-hit-rates.html", "published_at": "2026-10-02 00:00:00+00:00", "updated_at": "2026-10-03 04:08:36.332011+00:00", "lang": "en", "topics": ["developer-tools", "mlops"], "entities": ["CloudX", "cloudx-io/setup-go", "actions/setup-go", "GitHub Actions", "GOCACHE", "go list", "Go"], "also_reported_by": [], "alternates": {"html": "https://wpnews.pro/news/fast-counterfactuals-for-go-cache-hit-rates", "markdown": "https://wpnews.pro/news/fast-counterfactuals-for-go-cache-hit-rates.md", "text": "https://wpnews.pro/news/fast-counterfactuals-for-go-cache-hit-rates.txt", "jsonld": "https://wpnews.pro/news/fast-counterfactuals-for-go-cache-hit-rates.jsonld"}}