`gh attestation verify` said yes to an image we never released

A developer demonstrated a four-step container image verification workflow against the soit-ai/soit server v1.0.0 release, showing that fetching by digest, verifying provenance, reading the SBOM, and checking the release manifest are all required to confirm an image's authenticity. The walkthrough highlights that the OCI referrers API returned MANIFEST_UNKNOWN for both the index and platform manifest digests, and that the spec's tag-based fallback (sha256-<digest>) is needed to locate the Sigstore provenance and SPDX SBOM attestations. The author warns that skipping the release manifest check means verifying a different build than the one a version actually shipped.

Pulling a container image somebody else built asks four separate questions. Most people answer three of them and skip the fourth: | Step | Question it answers | If you skip it | |---|---|---| | Fetch by digest | are these the bytes I asked for | you are gambling on a mutable tag | | Verify provenance | who built these bytes, from what, where | unknown origin | | Read the SBOM | what is inside | a CVE lands and you cannot tell if you are affected | | Check the release manifest | is this digest the one that version shipped | you may be verifying a different build | The fourth is the one that has no substitute, and it is the one that is almost always skipped. What follows is all four, run for real against our own v1.0.0 . This is where the confusion starts. Get an anonymous pull token — a public package needs no account: TOKEN=$ curl -s "https://ghcr.io/token?scope=repository:soit-ai/soit/server:pull&service=ghcr.io" \ | sed -n 's/. "token":"\ ^" \ ". /\1/p' Ask what the v1.0.0 tag resolves to: curl -sI -H "Authorization: Bearer $TOKEN" \ -H "Accept: application/vnd.oci.image.index.v1+json" \ https://ghcr.io/v2/soit-ai/soit/server/manifests/v1.0.0 docker-content-digest: sha256:96b80ae141000adde27cf3dedb27935b5f5d0085d69025cffb4bbb63276d5929 Content-Type: application/vnd.oci.image.index.v1+json Content-Length: 856 856 bytes: this is an image index, not the image. Inside it, two entries: manifests 0 : sha256:84e7f539...d32f 2589 bytes linux/amd64 manifests 1 : sha256:a6bd430d...e1be 565 bytes unknown/unknown vnd.docker.reference.type: attestation-manifest vnd.docker.reference.digest: sha256:84e7f539...d32f That unknown/unknown entry is not corruption — it is BuildKit's own attestation for the amd64 manifest, and it is a different mechanism from the Sigstore signatures we are about to read. First thing people conflate. One level down, the amd64 manifest says: config: sha256:021814982fcc214a02a2b322cbc3729b3ce43bd6e191aa0fa39da540a3e3cd1f 8769 bytes layers: 12, 684216450 bytes compressed about 652 MiB Three hashes so far, three different meanings: | Name | Value for server v1.0.0 | Refers to | |---|---|---| | index digest | 96b80ae1…5929 | the multi-platform index itself | | platform manifest digest | 84e7f539…d32f | the linux/amd64 image | | config digest | 02181498…cd1f | the image config JSON | The signature covers the first one. A fourth hash shows up in section 8, and it is none of these. The OCI spec defines a referrers API for "things attached to this thing": curl -s -H "Authorization: Bearer $TOKEN" \ https://ghcr.io/v2/soit-ai/soit/server/referrers/sha256:96b80ae1...5929 {"errors": {"code":"MANIFEST UNKNOWN","message":"manifest unknown"} } Same error for the platform manifest digest. Do not conclude "unsigned" from this. The spec has a fallback: replace the colon with a dash and treat the digest {"name":"soit-ai/soit/server","tags": "v1.0.0", "sha256-3a5b3b1a2d14e0646298826602124ba22e63f8c078f653e506a0a042bfd18246", "sha256-236201a5a3ee861ffdb5fdd7ec134454619eb1ab0e777439c4a22a65002f874f", "sha256-96b80ae141000adde27cf3dedb27935b5f5d0085d69025cffb4bbb63276d5929" } The last one is ours. Remember the first two — section 9 is entirely about them. Fetching sha256-96b80ae1… returns a small index holding two Sigstore bundles: sha256:658b3428...80b9 812 bytes artifactType : application/vnd.dev.sigstore.bundle.v0.3+json predicateType: https://slsa.dev/provenance/v1 created : 2026-08-05T16:14:12.946Z sha256:59bac8c5...d9cf 814 bytes predicateType: https://spdx.dev/Document/v2.3 created : 2026-08-05T16:14:22.004Z Build provenance and an SBOM attestation, signed nine seconds apart. Both manifests carry subject pointing back at sha256:96b80ae1…5929 — the single place where "this signature belongs to that image" is actually written down. Pull the bundle blob and hash it: curl -sL -H "Authorization: Bearer $TOKEN" \ https://ghcr.io/v2/soit-ai/soit/server/blobs/sha256:eae5d902...dd86 -o bundle.json sha256sum bundle.json eae5d902186b490570aba6f702f7ce03c46f935e403bf881ec27d99480d7dd86 bundle.json The hash equals the digest I asked for. A content-addressed registry means this step verifies itself; no trust required yet. The bundle has three top-level fields — mediaType , verificationMaterial , dsseEnvelope . The envelope's payloadType is application/vnd.in-toto+json , and the base64 payload is the only thing the signature covers: { " type": "https://in-toto.io/Statement/v1", "subject": { "name": "ghcr.io/soit-ai/soit/server", "digest": {"sha256": "96b80ae141000adde27cf3dedb27935b5f5d0085d69025cffb4bbb63276d5929"} } , "predicateType": "https://slsa.dev/provenance/v1", "predicate": { "buildDefinition": { "buildType": "https://actions.github.io/buildtypes/workflow/v1", "externalParameters": { "workflow": { "ref": "refs/tags/v1.0.0", "repository": "https://github.com/soit-ai/soit", "path": ".github/workflows/release.yml" } }, "internalParameters": { "github": { "event name": "push", "repository id": "910429753", "repository owner id": "193298865", "runner environment": "github-hosted" } }, "resolvedDependencies": { "uri": "git+https://github.com/soit-ai/soit@refs/tags/v1.0.0", "digest": {"gitCommit": "8105cae074f1f27d7916acfe02f9d4eabb63169f"} } }, "runDetails": { "builder": {"id": "https://github.com/soit-ai/soit/.github/workflows/release.yml@refs/tags/v1.0.0"}, "metadata": {"invocationId": "https://github.com/soit-ai/soit/actions/runs/31023642816/attempts/1"} } } } One complete sentence: commit 8105cae… on refs/tags/v1.0.0 , built by .github/workflows/release.yml on a GitHub-hosted runner in run 31023642816, produced the image with digest 96b80ae1… . I had the repository on disk, so I closed the loop immediately: git rev-parse v1.0.0^{commit} 8105cae074f1f27d7916acfe02f9d4eabb63169f Match. Note repository id and repository owner id : numeric IDs are worth more than the repository name, because names can be renamed and transferred, IDs cannot. verificationMaterial.certificate.rawBytes is a 1735-byte DER certificate: issuer=O = sigstore.dev, CN = sigstore-intermediate subject= notBefore=Aug 5 16:14:11 2026 GMT notAfter =Aug 5 16:24:11 2026 GMT X509v3 Subject Alternative Name: critical URI:https://github.com/soit-ai/soit/.github/workflows/release.yml@refs/tags/v1.0.0 Two things worth stopping on. The subject is empty. The field that traditionally names the holder is blank; the identity moved entirely into that SAN URI, and it is not a person or an organisation — it is a workflow at a ref. The validity window is exactly ten minutes. That is keyless signing: the pipeline trades an OIDC token for a short-lived certificate, signs, and throws the key away. Nothing to store, nothing to rotate, nothing to steal later. Ten minutes on, the certificate is expired — and the signature still verifies, for the reason in section 5. The certificate also carries a block of Sigstore extensions describing the build. The values below are what I read; I am deliberately not putting names on the OIDs — those live in Fulcio's OID documentation, and the values speak for themselves: | OID suffix | Value | |---|---| | .1.1 / .1.8 | https://token.actions.githubusercontent.com | | .1.2 / .1.20 | push | | .1.3 / .1.10 / .1.13 / .1.19 | 8105cae074f1f27d7916acfe02f9d4eabb63169f | | .1.4 | release | | .1.5 | soit-ai/soit | | .1.6 / .1.14 | refs/tags/v1.0.0 | | .1.9 / .1.18 | …/release.yml@refs/tags/v1.0.0 | | .1.11 | github-hosted | | .1.12 | https://github.com/soit-ai/soit | | .1.15 / .1.17 | 910429753 / 193298865 | | .1.16 | https://github.com/soit-ai | | .1.21 | …/actions/runs/31023642816/attempts/1 | | .1.22 | public | | .1.24 | repo:soit-ai/soit:ref:refs/tags/v1.0.0 | Extended Key Usage is Code Signing , Key Usage is Digital Signature only, and there is a CT precertificate SCT timestamped Aug 5 16:14:11.958 2026 GMT — the act of issuing this certificate was itself logged publicly. verificationMaterial.tlogEntries holds exactly one record: php logIndex : 2346649359 integratedTime : 1785946452 - 2026-08-05T16:14:12Z kindVersion : {kind: dsse, version: 0.0.1} That record lives in Rekor, the public append-only transparency log, and its job is to witness when . The question a verifier asks is not "is this certificate valid now" — it expired long ago — but " was it valid at the moment of signing ". The log pins that moment at 16:14:12Z , inside the 16:14:11 – 16:24:11 window. A detail from the actual bundle: timestampVerificationData is an empty object . No RFC 3161 timestamp; the Rekor entry carries the time on its own. Not wrong, just worth knowing what your trust actually rests on. The second bundle's predicate is a complete SPDX document. The bundle blob is 3437645 bytes: spdxVersion : SPDX-2.3 dataLicense : CC0-1.0 name : ghcr.io/soit-ai/soit/server creationInfo : creators "Organization: Anchore, Inc", "Tool: syft-1.42.3" , created "2026-08-05T16:14:08Z" packages : 710 relationships : 2840 Those 710 packages by purl type: 469 pkg:deb , 217 pkg:pypi , one pkg:generic python 3.11.15 and one pkg:oci the image itself . The 469 Debian packages come from the base image — server/Dockerfile starts at python:3.11 — and only 217 are our own Python dependencies. That ratio is itself a finding: two thirds of what you just signed, you did not write. All three images side by side show the split clearly: | Image | Packages | Composition | SBOM generated | |---|---|---|---| | server | 710 | 469 deb + 217 pypi | 16:14:08Z | | knowledge-worker | 775 | 469 deb + 281 pypi | 16:22:23Z | | web | 1331 | 1311 npm + 18 apk | 16:15:01Z | The 64 extra pypi packages in knowledge-worker include torch , transformers and nvidia-cublas-cu12 . The two images are two targets of one Dockerfile; the entire difference is uv sync --extra knowledge-worker server/Dockerfile:19 vs :27 . web is node:24-alpine , so the Debian packages become 18 apk ones. An SPDX document normally carries a files section recording which files each package was identified from. All three of ours have none — and it is deliberate, right there in the workflow .github/workflows/release.yml:132–149 : jq 'del .files | .packages |= map del .hasFiles | .relationships |= map select .spdxElementId // "" | startswith "SPDXRef-File" | not and .relatedSpdxElement // "" | startswith "SPDXRef-File" | not ' \ "$f" "$f.tmp" mv "$f.tmp" "$f" size="$ stat -c%s "$f" " test "$size" -le 16000000 The comment above it gives the reason: per-file SPDX entries push an ML-heavy image's SBOM past the 16 MiB subject limit of actions/attest . In plain terms, the SBOM had to be shrunk until it could be signed. The cost is real. You keep the package inventory and the package-to-package relationships; you lose "which file did this package come from", which is exactly what you want when a CVE lands and you need to locate it. The test -le 16000000 line is a hard gate: if a future image blows past the ceiling the release fails there rather than shipping an SBOM that cannot be signed. That part I think is right — better a broken build than a half-signed artifact. Two things I found in ours. Five Windows launchers in a Linux image. Among the 710 packages, Simple Launcher 1.1.0.14 appears five times , each with a CPE and no purl: SPDXRef-Package-binary-Simple-Launcher-d60858f6579e7bb1 versionInfo : 1.1.0.14 externalRefs: cpe:2.3:a:Simple Launcher:Simple Launcher:1.1.0.14: : : : : : : Syft's binary classifier found them: Python packaging wheels ship small Windows launcher executables, which will never execute inside this Linux image but are inventoried all the same. And because the files section was stripped section 7 , you cannot even find out where they are. A fourth hash that matches nothing. The pkg:oci entry representing the image itself reads: pkg:oci/ghcr.io%2Fsoit-ai%2Fsoit%2Fserver@sha256%3Ae7c993b9ac5d7322058e0169c678b4ce21fe42fac72c5d3374404bf4642939ea?arch=amd64 e7c993b9… is not the index digest, not the platform manifest digest and not the config digest. Asked for directly, the registry returns 404. I could not determine what it is — the reasonable guess is an identifier the scanner computes locally — so I am reporting the observation and not a conclusion. The practical rule is clear enough: do not reconcile an SBOM's internal digest against the signature's subject. The binding lives in exactly one place, subject.digest , and that value is 96b80ae1… , matching the registry exactly. v1.0.0 images, all of them verifiable Back to those two extra tags. Fetched the same way: | Image digest behind the fallback tag | Attestations attached | Signed at | |---|---|---| | 3a5b3b1a…8246 | provenance only | 2026-08-05T15:41:15Z | | 236201a5…f874f | provenance + SBOM | 15:52:32Z / 15:52:44Z | | 96b80ae1…5929 | provenance + SBOM | 16:14:12Z / 16:14:22Z | All three provenance statements name refs/tags/v1.0.0 and the same signing identity. The commits differ: | Image digest | Commit in provenance | Actions run | Commit subject | |---|---|---|---| | 3a5b3b1a… | 0dacfc52… | 31020921135 | ci release : create the artifacts directory before image SBOM generation | | 236201a5… | ec822c63… | 31021873557 | ci release : catalog packages only in image SBOMs | | 96b80ae1… | 8105cae0… | 31023642816 | ci release : trim image SBOMs to package level before attestation | All three commits are on main locally, 23:33 / 23:44 / 00:05 in my timezone. The story reads itself off the commit subjects: the tag was re-pointed twice that night while the SBOM step was being fixed — and each failed attempt had already So this happens: gh attestation verify \ oci://ghcr.io/soit-ai/soit/server@sha256:3a5b3b1a...8246 \ --repo soit-ai/soit --format json exit code : 0 subject : 3a5b3b1a...8246 predicate : https://slsa.dev/provenance/v1 ref : refs/tags/v1.0.0 commit : 0dacfc5219c4ae57d024f9cb4208e1efad5f0d17 san : https://github.com/soit-ai/soit/.github/workflows/release.yml@refs/tags/v1.0.0 tlog : rekor.sigstore.dev @ 2026-08-05T23:41:11+08:00 Exit 0. Nothing is wrong with the signature. It states honestly what it is: an image built by our release workflow, on refs/tags/v1.0.0 , from commit 0dacfc52… . The one thing it does not say is that it is not the image we shipped. No attack here — we built all three. But swap in a nastier scenario and it holds immediately: a pipeline compromised once, fixed, re-tagged, with the poisoned intermediate still sitting in the registry. Attestation will not catch that. gh attestation verify answers "did this come out of that pipeline" . It never promised to answer "is this the release" . That answer is not in the signature. It is an asset on the GitHub Release, which carries seven files: soit-1.0.0.tar.gz 3148831 SHA256SUMS 433 release-artifacts.json 2877 server.spdx.json 3313874 knowledge-worker.spdx.json 4004516 web.spdx.json 3862614 source.spdx.json 3694033 release-artifacts.json is the manifest: { "featureKey": "release.artifacts", "schemaVersion": 1, "version": "1.0.0", "release tag": "v1.0.0", "commit": "8105cae074f1f27d7916acfe02f9d4eabb63169f", "clean worktree at tag": true, "images": { "component": "server", "name": "ghcr.io/soit-ai/soit/server", "release tag": "v1.0.0", "digest": "sha256:96b80ae141000adde27cf3dedb27935b5f5d0085d69025cffb4bbb63276d5929", "reference": "ghcr.io/soit-ai/soit/server@sha256:96b80ae1...5929", "sbom": {"format": "spdx-json", "path": "server.spdx.json", "sha256": "b83c3bc7...ef68", "attestation": "..."}, "provenance attestation": "..." } } It says the thing the signature would not: v1.0.0 is 96b80ae1… , not the other two. It is assembled by jq inside the publish-release job release.yml:300–347 and validated on the spot. I fed the copy downloaded from the public internet straight into the same script that lives in the repository: { "commit": "8105cae074f1f27d7916acfe02f9d4eabb63169f", "images": "knowledge-worker", "server", "web" , "passed": true, "release tag": "v1.0.0" } exit=0 178 lines, standard library only, no network server/scripts/verify release artifacts.py . It insists that: release tag equals v + version :42 ; commit is 40 lowercase hex characters and not all zeroes :45 ; clean worktree at tag is true :49 ; reference name@digest :91 — a name:tag reference is rejected outright; digest matches ^sha256: 0-9a-f {64}$ :88 ; :15 , :114 ; :107–112 . A test guards that last-but-two rule: it rewrites the example's reference into tag form and asserts the script fails with digest-pinned server/tests/unit/test release operations contract.py:85–88 . The release also ships a git archive tarball release.yml:253–263 described as deterministic. Worth testing rather than believing. First, the checksum: grep "soit-1.0.0.tar.gz" SHA256SUMS | sha256sum -c - ./soit-1.0.0.tar.gz: OK Then rebuild it locally: git archive --format=tar.gz --prefix="soit-1.0.0/" \ --output=local.tar.gz 8105cae074f1f27d7916acfe02f9d4eabb63169f c11179c379ba7390c215133f342c92a7517a548f1536959cb43e187b16a7bab3 local.tar.gz 5d3dd50f491a897ba9a184ad6dc474e2ef55508224404266d91b3e1e1d32ae9d soit-1.0.0.tar.gz Different — and not just at the gzip layer; the inner tars differ too. Comparing members explains it: members : 1551 local / 1551 released -- same only-one-side: 0 / 0 -- same differing : 1280 members mtime : 1785945959 everywhere -- same mode : 0664 everywhere -- same example: soit-1.0.0/.github/workflows/quality.yml local 15270 released 14803 Sizes only, and each delta equals that file's line count. CRLF. My machine has core.autocrlf=true , so git archive helpfully rewrote the line endings. Turn it off: git -c core.autocrlf=false archive --format=tar.gz --prefix="soit-1.0.0/" \ --output=local2.tar.gz 8105cae074f1f27d7916acfe02f9d4eabb63169f 5d3dd50f491a897ba9a184ad6dc474e2ef55508224404266d91b3e1e1d32ae9d local2.tar.gz Identical. A Windows box, git 2.55.0, a month later, reproducing the tarball GitHub Actions produced. Both halves matter. Determinism is real — git archive writes no build timestamp and takes mtimes from the commit — but it is sensitive to local git configuration , and the first suspect when reproduction fails is core.autocrlf , not the publisher. Everything above, compressed into something you can paste: 1. Resolve the digest never keep books against a tag docker buildx imagetools inspect ghcr.io/soit-ai/soit/server:v1.0.0 | head -3 2. Verify build provenance gh attestation verify oci://ghcr.io/soit-ai/soit/server:v1.0.0 --repo soit-ai/soit 3. Verify the SBOM attestation NOT checked by default gh attestation verify oci://ghcr.io/soit-ai/soit/server:v1.0.0 --repo soit-ai/soit \ --predicate-type https://spdx.dev/Document/v2.3 4. Check the release manifest: is this digest the one that shipped curl -sLO https://github.com/soit-ai/soit/releases/download/v1.0.0/release-artifacts.json curl -sLO https://github.com/soit-ai/soit/releases/download/v1.0.0/SHA256SUMS sha256sum -c SHA256SUMS 5. Run it by digest, not by tag docker pull ghcr.io/soit-ai/soit/server@sha256:96b80ae1...5929 Step 3 deserves emphasis: gh attestation verify checks provenance only by default. Without --predicate-type it returned exactly one result here, the SLSA provenance. With it, the SPDX attestation comes back — 6022876 bytes of JSON holding the same 710 packages counted in section 6. A lot of people assume "verified" includes the SBOM. It does not. Step 4 is the point of this whole piece. Without it, the green checkmark from step 2 shines just as brightly on the 3a5b3b1a… image from section 9. House rule: the last section is about our own problems. 1. Our own compose pins tags, not digests. All six services in docker/docker-compose.images.yml use :${SOIT IMAGE TAG:-v1.0.0} :17 – :32 , and the README says the same README.md:146 . We demand digest-pinned references inside release-artifacts.json and then hand users a tag. Impact: a tag can be re-pointed. Workaround: step 5 of the checklist. I plan to open an issue ; none filed at the time of writing, hence no link. 2. SHA256SUMS does not cover release-artifacts.json . The 433-byte file lists five entries — four SPDX documents and the tarball. Neither the manifest nor SHA256SUMS itself is in there the latter cannot be, by construction . The manifest does get its own attestation release.yml:350–353 , so it is not unprotected, but the instinctive "download everything and run sha256sum -c " does not reach it. Also going to open an issue. 3. docs/release-process.md lags the pipeline. It tells you to copy the real tag, commit, digests, SBOM checksums and attestation URLs into an evidence document and then run the verifier :24 – :32 . The pipeline has been doing that automatically for a while, and uploads the result as a release asset. Impact: a reader concludes the manifest is hand-written after the fact, which makes it look weaker than it is. Also going to open an issue. 4. Superseded builds are never cleaned up. The two images from section 9 are still pullable, still verifiable, still runnable. We have no "delete same-tag intermediates after a successful release" step. Impact: as described. Workaround: pull the digest from release-artifacts.json . 5. The verifier checks shape, not truth. provenance attestation only has to be a non-empty string :58 , :64 , :106 , all via require text . The script never fetches the URL and never compares the attested subject digest to the one in the manifest. Impact: a well-formed manifest full of fabricated URLs passes. Workaround: it is a structural check, not a trust check — the trust comes from gh attestation verify , and you need both. Defensible division of labour, but the docs never say so. 6. REQUIRED IMAGES is hard-coded. verify release artifacts.py:15 pins {"server", "knowledge-worker", "web"} and line 114 demands exact equality. Add a fourth image and the release breaks at the final step. Impact: a bad surprise on release day. Workaround: know it is there — and the case it blocks silently shipping with an image missing is worth more than the nuisance. 7. No process for SBOM false positives. Sections 7 and 8. Nobody owns annotating Simple Launcher as noise, so the next reader has to work it out again. Impact: the signal-to-noise ratio degrades as images grow. 8. I could not identify e7c993b9… . That is where section 8 stops. This entry exists so I do not pretend otherwise : I can confirm it is absent from the registry 8105cae… . Later releases will have different values; the method does not change. gh attestation verify A passing attestation says "these bytes came out of that pipeline"; it does not say "this is the release." What joins the two is a manifest binding tag, commit and release-artifacts.json , watched by a 178-line script whose hardest The repository is github.com/soit-ai/soit , and every step above is reproducible: curl calls in section 2 need no account — an anonymous token is enough; 3a5b3b1a… image from section 9 is still there; verify it yourself and watch it exit 0; -c core.autocrlf=false . If any claim here is wrong — especially the digest in section 8 that I failed to identify — open an issue and say so. I would rather learn where this is wrong than be told the pipeline looks thorough.