Poisoning a Go Cache

A blog post by Lukas Schwab demonstrates that Go's build cache ($GOCACHE) can be poisoned to reverse a program's behavior without changing its source code, using a minimal demo module where `println(truthy.Value())` prints `false` instead of `true`. The post explains that the cache stores content-addressed `-d` data files, whose filenames are the SHA-256 sum of their contents, alongside `-a` action files that map action IDs to outputs. The demonstration follows an earlier CloudX post on scaling Go CI by replacing `actions/setup-go`.

Earlier this month I explained Go’s caches in a post I coauthored for CloudX, ‘Scaling Golang CI by Replacing actions/setup-go .’ https://www.cloudx.ai/posts/setup-go If you run Go in GitHub Actions, the post deserves a read and cloudx-io/setup-go deserves your consideration . If you just want to learn more about Go caches without thinking about GitHub, this post is for you. Your Go cache $GOCACHE is a directory of files mapping “action IDs” — hashes representing an atomic part of a request, such as a build — to corresponding outputs. At CloudX we use this understanding to make our engineers more effective, but you can use the same understanding to do surprising, stupid, pathological things instead. This post will explore your Go cache’s structure in greater depth by demonstrating cache poisoning https://en.wikipedia.org/wiki/Cache poisoning : we can change the behavior of a Go program by manipulating the cache, even without changing the program’s source code. We’ll use an absolutely minimal demo project, but the same principles apply to much more complex Go modules. Our demo is a simple Go module: It contains a minimal go.mod : module github.com/lukasschwab/demo go 1.27 It contains a target package truthy , which just provides the boolean value true : package truthy func Value bool { return true } Finally, it contains an entry-point in main.go for demonstration purposes: package main import "github.com/lukasschwab/demo/truthy" func main { println truthy.Value } What happens if you run main.go ? Exactly what you’d expect: bash $ go run main.go true Well, that’s what it does if nobody has poisoned your Go cache. Without changing this source code in any way, we can reverse the program’s behavior: println truthy.Value will print false First, observe the cache contents. The Go toolchain populates the cache once, then reuses the cache contents for later requests. We’ll poison the cached artifacts for package truthy ; let’s start by building it with an isolated cache so we can inspect these artifacts. bash $ GOCACHE="/demo/cache" go build ./truthy $ tree --prune /demo/cache /demo/cache ├── 25 │ └── 25b28cbb223b28161c768050739328f6f0fdc5e83a2618046ff89448f3a6c708-d ├── 2f │ └── 2f8abba195240b3d4e97965b71d12242740c4bae769f56618902ca1cb8e45c17-d ├── 56 │ └── 56394f70c8c6d21aa0d64a9ab4c8dec7cd2f2f864a9577baf5d5eceafd1d76c6-a ├── 7a │ └── 7ad9f061c611f521651194d122993f9a9226f606e05e76ac0e1eafe161ec1091-d ├── 9b │ └── 9b6e194bae01948fba16d0d48981c453d8ce937c45333f1888348f6ec67efdf6-a ├── ac │ └── ac9c6630f8e0cb23af031113e331e8a2c5ff0ed127b536593c2e7abe3c055d43-a ├── c2 │ └── c22e5acea0c87a8cdbc9eb8d5195bde41f2bb41605c29622c82b6cee553f6b38-a ├── e3 │ └── e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855-d ├── README └── trim.txt There are two kinds of files in the cache, distinguished by suffix: -d and -a . -d files are data; the whole point of the cache is to make these outputs reusable. Specifically, in our example these are package archives: portable, intermediate build artifacts that can be linked together into executable binaries. These files are content-addressed — the filename is always the SHA-256 sum of the file contents: bash $ sha256sum demo/cache/e3/e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855-d e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855 /demo/cache/e3/e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855-d Content-addressing data this way compacts the cache. We could have seven independent packages that all depend on package truthy ; if they share a Go cache, it will only store package truthy ’s data once. Taken alone, these -d files are opaque. To derive the filename you want to restore, you need to compute the file contents; that defeats the purpose of caching them This is where the -a files come in: these are action indexes, and they map action identifiers — hashes of user inputs, including source code, flags, dependencies, and environment variables — to the corresponding package archives. Each -a file contains a single line with five whitespace-separated values. Take a peek inside one of the action indexes for package truthy , 56394f7…-a : | Value | Meaning | |---|---| | v1 | Format version — always v1 for now. | | 56394f7… | The action identifier, matching this -a filename. | | 2f8abba… | The output identifier, pointing to a 2f8abba…-d data file in the cache. | | 2218 | Size of the 2f8abba…-d data file in bytes. | | 1790143015118664000 | Unix timestamp for this action index nanoseconds . | The build cache saves time because action IDs are much cheaper to calculate than the package archives stored in data files. To use the cache, the Go toolchain calculates an action ID and checks if it’s present in the cache. If there’s a corresponding -a file, Go follows the pointer to the -d data and reuses it — job done. Otherwise, it proceeds to build the package archive expedited by cache checks deeper in the build tree before stashing the final data in a -d - and -a -file pair. To poison our program’s behavior, we need to intercept this cache-lookup process. Specifically, we can find the true action ID for the truthy package build, but rewrite its -a file to point at a poisonous new -d package archive. To get the action ID for package truthy , work backwards: use go list to find its package archive, then find the action index file pointing at it. bash Find the final package archive data for package truthy. $ GOCACHE="/demo/cache" go list -export -f '{{.Export}}' ./truthy /demo/cache/2f/2f8abba195240b3d4e97965b71d12242740c4bae769f56618902ca1cb8e45c17-d Use that hash to find the corresponding action index. $ grep -l "2f8abba195240b3d4e97965b71d12242740c4bae769f56618902ca1cb8e45c17" \ /demo/cache/ / -a cache/56/56394f70c8c6d21aa0d64a9ab4c8dec7cd2f2f864a9577baf5d5eceafd1d76c6-a The other files in the cache, under other hashes, represent other intermediates in the go build process. In theory those could also be targets for poisoning, but for the purposes of this demo they’re superfluous. Define a poison file with exactly the same public interface as truthy.go . Sharing this external interface ensures the poison package archive will link into the same binaries the target truthy package would. package truthy func Value bool { return false } We also need Go to build our poison package archive as if it corresponded to /demo/truthy rather than /demo/poison . To do this, we use a go tool feature called “overlays.” Anytime the Go build tools would read truthy/truthy.go , this overlay replaces that with a read of poison/truthy.go : { "Replace": { "/demo/truthy/truthy.go": "demo/poison/truthy.go" } } So now, when we build package truthy with that overlay, we’ll write a poisoned package archive into the cache: bash $ GOCACHE="/demo/cache" go list -overlay overlay.json -export -f '{{.Export}}' \ ./truthy /demo/cache/2d/2dbc3cb27e5eb61aa8267224780ecf293e31ecedc587ba0752ed07d6334abf2e-d The last step is to update the -a package index we identified for truthy earlier with a reference to this poisoned package archive: replace the healthy hash 2f8abba… with the poisoned hash 2dbc3cb… and update the stored data file size to match in this case, it’s unchanged . bash $ cat cache/56/56394f70c8c6d21aa0d64a9ab4c8dec7cd2f2f864a9577baf5d5eceafd1d76c6-a v1 56394f70c8c6d21aa0d64a9ab4c8dec7cd2f2f864a9577baf5d5eceafd1d76c6 2f8abba195240b3d4e97965b71d12242740c4bae769f56618902ca1cb8e45c17 2218 1790143015118664000 Rewrite the healthy output identifier with our poisoned output identifier. $ sed -i '' 's/2f8abba195240b3d4e97965b71d12242740c4bae769f56618902ca1cb8e45c17/2dbc3cb27e5eb61aa8267224780ecf293e31ecedc587ba0752ed07d6334abf2e/g' cache/56/56394f70c8c6d21aa0d64a9ab4c8dec7cd2f2f864a9577baf5d5eceafd1d76c6-a $ cat cache/56/56394f70c8c6d21aa0d64a9ab4c8dec7cd2f2f864a9577baf5d5eceafd1d76c6-a v1 56394f70c8c6d21aa0d64a9ab4c8dec7cd2f2f864a9577baf5d5eceafd1d76c6 2dbc3cb27e5eb61aa8267224780ecf293e31ecedc587ba0752ed07d6334abf2e 2218 1790143015118664000 Now running our entry-point script triggers a build that blithely uses the poisoned package archive: bash We've already poisoned the cache, so we can delete the source files. $ rm -rf ./poison The source code calls ./truthy, but this run exhibits poisoned behavior $ GOCACHE="/demo/cache" go run main.go false Of course, this example is contrived. Cache poisoning in anger would look very different: truthy , replacing common dependencies like the Go standard library’s crypto with malicious alternates. go list cache inspections we did here can be precomputed for common OS architectures, Go versions, etc. Aside from those parameters, nothing about your Go cache is unique to your device. An adversary with write access to files on your machine has better things to do than tamper with your Go cache — by rational choice, they’ll steal your secret keys instead. Cache poisoning is a more pressing concern when it smuggles logic over a trust boundary. That’s why GitHub’s Actions cache allows non-default branches to read cache entries produced by the default, but prevents the default from ever loading a cache entry produced by a non-default branch: an untrusted PR could poison the Actions cache with a package archive that steals credentials when executed in the privileged default-branch context. If you’ve ever wondered why your Go tests always re-execute on main even though they just passed in your PR checks, now you know I’d have a hard time spotting cache poisoning in CI. You could push code to a PR that poisons the cache, then force-push to eliminate that commit. Even though it the poisoning code won’t appear in the commit history, later CI runs will blithely use the malicious records. If you can’t trust everything with access to your build cache, you can’t trust the builds downstream.