Poisoning a Go Cache

Earlier this month I explained Go’s caches in a post I coauthored for CloudX, ‘Scaling Golang CI by Replacing actions/setup-go.’ If you run Go in GitHub Actions, the post deserves a read (and cloudx-io/setup-go deserves your consideration). If you just want to learn more about Go caches without thinking about GitHub, this post is for you.

Your Go cache ($GOCACHE) is a directory of files mapping “action IDs” — hashes representing an atomic part of a request, such as a build — to corresponding outputs. At CloudX we use this understanding to make our engineers more effective, but you can use the same understanding to do surprising, stupid, pathological things instead.

This post will explore your Go cache’s structure in greater depth by demonstrating cache poisoning: we can change the behavior of a Go program by manipulating the cache, even without changing the program’s source code.

We’ll use an absolutely minimal demo project, but the same principles apply to much more complex Go modules. Our demo is a simple Go module:

It contains a minimal go.mod:

module github.com/lukasschwab/demo

go 1.27

It contains a target package truthy, which just provides the boolean value true:

package truthy

func Value() bool { return true }

Finally, it contains an entry-point in main.go for demonstration purposes:

package main

import "github.com/lukasschwab/demo/truthy"

func main() {
    println(truthy.Value())
}

What happens if you run main.go? Exactly what you’d expect:

$ go run main.go
true

Well, that’s what it does if nobody has poisoned your Go cache. Without changing this source code in any way, we can reverse the program’s behavior: println(truthy.Value()) will print false!

First, observe the cache contents. The Go toolchain populates the cache once, then reuses the cache contents for later requests. We’ll poison the cached artifacts for package truthy; let’s start by building it with an isolated cache so we can inspect these artifacts.

$ GOCACHE="/demo/cache" go build ./truthy
$ tree --prune /demo/cache
/demo/cache
├── 25
│   └── 25b28cbb223b28161c768050739328f6f0fdc5e83a2618046ff89448f3a6c708-d
├── 2f
│   └── 2f8abba195240b3d4e97965b71d12242740c4bae769f56618902ca1cb8e45c17-d
├── 56
│   └── 56394f70c8c6d21aa0d64a9ab4c8dec7cd2f2f864a9577baf5d5eceafd1d76c6-a
├── 7a
│   └── 7ad9f061c611f521651194d122993f9a9226f606e05e76ac0e1eafe161ec1091-d
├── 9b
│   └── 9b6e194bae01948fba16d0d48981c453d8ce937c45333f1888348f6ec67efdf6-a
├── ac
│   └── ac9c6630f8e0cb23af031113e331e8a2c5ff0ed127b536593c2e7abe3c055d43-a
├── c2
│   └── c22e5acea0c87a8cdbc9eb8d5195bde41f2bb41605c29622c82b6cee553f6b38-a
├── e3
│   └── e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855-d
├── README
└── trim.txt

There are two kinds of files in the cache, distinguished by suffix: -d and -a.

-d files are data; the whole point of the cache is to make these outputs reusable. Specifically, in our example these are package archives: portable, intermediate build artifacts that can be linked together into executable binaries. These files are content-addressed — the filename is always the SHA-256 sum of the file contents:

$ sha256sum demo/cache/e3/e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855-d
e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855  /demo/cache/e3/e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855-d

Content-addressing data this way compacts the cache. We could have seven independent packages that all depend on package truthy; if they share a Go cache, it will only store package truthy’s data once.

Taken alone, these -d files are opaque. (To derive the filename you want to restore, you need to compute the file contents; that defeats the purpose of caching them!) This is where the -a files come in: these are action indexes, and they map action identifiers — hashes of user inputs, including source code, flags, dependencies, and environment variables — to the corresponding package archives.

Each -a file contains a single line with five whitespace-separated values. Take a peek inside one of the action indexes for package truthy, 56394f7…-a:

The build cache saves time because action IDs are much cheaper to calculate than the package archives stored in data files. To use the cache, the Go toolchain calculates an action ID and checks if it’s present in the cache. If there’s a corresponding -a file, Go follows the pointer to the -d data and reuses it — job done. Otherwise, it proceeds to build the package archive (expedited by cache checks deeper in the build tree) before stashing the final data in a -d- and -a-file pair.

To poison our program’s behavior, we need to intercept this cache-lookup process. Specifically, we can find the true action ID for the truthy package build, but rewrite its -a file to point at a poisonous new -d package archive.

To get the action ID for package truthy, work backwards: use go list to find its package archive, then find the action index file pointing at it.

$ GOCACHE="/demo/cache" go list -export -f '{{.Export}}' ./truthy
/demo/cache/2f/2f8abba195240b3d4e97965b71d12242740c4bae769f56618902ca1cb8e45c17-d

$ grep -l "2f8abba195240b3d4e97965b71d12242740c4bae769f56618902ca1cb8e45c17" \
    /demo/cache/**/*-a
cache/56/56394f70c8c6d21aa0d64a9ab4c8dec7cd2f2f864a9577baf5d5eceafd1d76c6-a

The other files in the cache, under other hashes, represent other intermediates in the go build process. In theory those could also be targets for poisoning, but for the purposes of this demo they’re superfluous.

Define a poison file with exactly the same public interface as truthy.go. Sharing this external interface ensures the poison package archive will link into the same binaries the target truthy package would.

package truthy

func Value() bool { return false }

We also need Go to build our poison package archive as if it corresponded to /demo/truthy rather than /demo/poison. To do this, we use a go tool feature called “overlays.” Anytime the Go build tools would read truthy/truthy.go, this overlay replaces that with a read of poison/truthy.go:

{
    "Replace": {
        "/demo/truthy/truthy.go": "demo/poison/truthy.go"
    }
}

So now, when we build package truthy with that overlay, we’ll write a poisoned package archive into the cache:

$ GOCACHE="/demo/cache" go list -overlay overlay.json -export -f '{{.Export}}' \
    ./truthy
/demo/cache/2d/2dbc3cb27e5eb61aa8267224780ecf293e31ecedc587ba0752ed07d6334abf2e-d

The last step is to update the -a package index we identified for truthy earlier with a reference to this poisoned package archive: replace the healthy hash 2f8abba… with the poisoned hash 2dbc3cb… and update the stored data file size to match (in this case, it’s unchanged).

$ cat cache/56/56394f70c8c6d21aa0d64a9ab4c8dec7cd2f2f864a9577baf5d5eceafd1d76c6-a
v1 56394f70c8c6d21aa0d64a9ab4c8dec7cd2f2f864a9577baf5d5eceafd1d76c6 2f8abba195240b3d4e97965b71d12242740c4bae769f56618902ca1cb8e45c17                 2218  1790143015118664000

$ sed -i '' 's/2f8abba195240b3d4e97965b71d12242740c4bae769f56618902ca1cb8e45c17/2dbc3cb27e5eb61aa8267224780ecf293e31ecedc587ba0752ed07d6334abf2e/g' cache/56/56394f70c8c6d21aa0d64a9ab4c8dec7cd2f2f864a9577baf5d5eceafd1d76c6-a
$ cat cache/56/56394f70c8c6d21aa0d64a9ab4c8dec7cd2f2f864a9577baf5d5eceafd1d76c6-a
v1 56394f70c8c6d21aa0d64a9ab4c8dec7cd2f2f864a9577baf5d5eceafd1d76c6 2dbc3cb27e5eb61aa8267224780ecf293e31ecedc587ba0752ed07d6334abf2e                 2218  1790143015118664000

Now running our entry-point script triggers a build that blithely uses the poisoned package archive:

$ rm -rf ./poison

$ GOCACHE="/demo/cache" go run main.go
false

Of course, this example is contrived. Cache poisoning in anger would look very different:

truthy, replacing common dependencies like the Go standard library’s crypto with malicious alternates.go list cache inspections we did here can be precomputed for common OS architectures, Go versions, etc. Aside from those parameters, nothing about your Go cache is unique to your device. An adversary with write access to files on your machine has better things to do than tamper with your Go cache — by rational choice, they’ll steal your secret keys instead.

Cache poisoning is a more pressing concern when it smuggles logic over a trust boundary. That’s why GitHub’s Actions cache allows non-default branches to read cache entries produced by the default, but prevents the default from ever a cache entry produced by a non-default branch: an untrusted PR could poison the Actions cache with a package archive that steals credentials when executed in the privileged default-branch context. If you’ve ever wondered why your Go tests always re-execute on main even though they just passed in your PR checks, now you know!

I’d have a hard time spotting cache poisoning in CI. You could push code to a PR that poisons the cache, then force-push to eliminate that commit. Even though it the poisoning code won’t appear in the commit history, later CI runs will blithely use the malicious records.

If you can’t trust everything with access to your build cache, you can’t trust the builds downstream.