# `gh attestation verify` said yes to an image we never released > Source: > Published: 2026-09-13 22:38:41+00:00 Pulling a container image somebody else built asks four separate questions. Most people answer three of them and skip the fourth: | Step | Question it answers | If you skip it | |---|---|---| | Fetch by digest | are these the bytes I asked for | you are gambling on a mutable tag | | Verify provenance | who built these bytes, from what, where | unknown origin | | Read the SBOM | what is inside | a CVE lands and you cannot tell if you are affected | | Check the release manifest | **is this digest the one that version shipped** | you may be verifying a different build | The fourth is the one that has no substitute, and it is the one that is almost always skipped. What follows is all four, run for real against our own `v1.0.0`. This is where the confusion starts. Get an anonymous pull token — a public package needs no account: ``` TOKEN=$(curl -s "https://ghcr.io/token?scope=repository:soit-ai/soit/server:pull&service=ghcr.io" \ | sed -n 's/.*"token":"\([^"]*\)".*/\1/p') ``` Ask what the `v1.0.0` tag resolves to: ``` curl -sI -H "Authorization: Bearer $TOKEN" \ -H "Accept: application/vnd.oci.image.index.v1+json" \ https://ghcr.io/v2/soit-ai/soit/server/manifests/v1.0.0 docker-content-digest: sha256:96b80ae141000adde27cf3dedb27935b5f5d0085d69025cffb4bbb63276d5929 Content-Type: application/vnd.oci.image.index.v1+json Content-Length: 856 ``` 856 bytes: this is an image index, not the image. Inside it, two entries: ``` manifests[0]: sha256:84e7f539...d32f 2589 bytes linux/amd64 manifests[1]: sha256:a6bd430d...e1be 565 bytes unknown/unknown vnd.docker.reference.type: attestation-manifest vnd.docker.reference.digest: sha256:84e7f539...d32f ``` That `unknown/unknown` entry is not corruption — it is BuildKit's own attestation for the amd64 manifest, and it is **a different mechanism from the Sigstore signatures** we are about to read. First thing people conflate. One level down, the amd64 manifest says: ``` config: sha256:021814982fcc214a02a2b322cbc3729b3ce43bd6e191aa0fa39da540a3e3cd1f 8769 bytes layers: 12, 684216450 bytes compressed (about 652 MiB) ``` Three hashes so far, three different meanings: | Name | Value for server v1.0.0 | Refers to | |---|---|---| | index digest | `96b80ae1…5929` | the multi-platform index itself | | platform manifest digest | `84e7f539…d32f` | the linux/amd64 image | | config digest | `02181498…cd1f` | the image config JSON | **The signature covers the first one.** A fourth hash shows up in section 8, and it is none of these. The OCI spec defines a referrers API for "things attached to this thing": ``` curl -s -H "Authorization: Bearer $TOKEN" \ https://ghcr.io/v2/soit-ai/soit/server/referrers/sha256:96b80ae1...5929 {"errors":[{"code":"MANIFEST_UNKNOWN","message":"manifest unknown"}]} ``` Same error for the platform manifest digest. **Do not conclude "unsigned" from this.** The spec has a fallback: replace the colon with a dash and treat the digest ``` {"name":"soit-ai/soit/server","tags":[ "v1.0.0", "sha256-3a5b3b1a2d14e0646298826602124ba22e63f8c078f653e506a0a042bfd18246", "sha256-236201a5a3ee861ffdb5fdd7ec134454619eb1ab0e777439c4a22a65002f874f", "sha256-96b80ae141000adde27cf3dedb27935b5f5d0085d69025cffb4bbb63276d5929"]} ``` The last one is ours. **Remember the first two — section 9 is entirely about them.** Fetching `sha256-96b80ae1…` returns a small index holding two Sigstore bundles: ``` sha256:658b3428...80b9 812 bytes artifactType : application/vnd.dev.sigstore.bundle.v0.3+json predicateType: https://slsa.dev/provenance/v1 created : 2026-08-05T16:14:12.946Z sha256:59bac8c5...d9cf 814 bytes predicateType: https://spdx.dev/Document/v2.3 created : 2026-08-05T16:14:22.004Z ``` Build provenance and an SBOM attestation, signed nine seconds apart. Both manifests carry `subject` pointing back at `sha256:96b80ae1…5929` — **the single place where "this signature belongs to that image" is actually written down.** Pull the bundle blob and hash it: ``` curl -sL -H "Authorization: Bearer $TOKEN" \ https://ghcr.io/v2/soit-ai/soit/server/blobs/sha256:eae5d902...dd86 -o bundle.json sha256sum bundle.json # eae5d902186b490570aba6f702f7ce03c46f935e403bf881ec27d99480d7dd86 *bundle.json ``` The hash equals the digest I asked for. A content-addressed registry means this step verifies itself; no trust required yet. The bundle has three top-level fields — `mediaType`, `verificationMaterial`, `dsseEnvelope`. The envelope's `payloadType` is `application/vnd.in-toto+json`, and the base64 payload is the only thing the signature covers: ``` { "_type": "https://in-toto.io/Statement/v1", "subject": [{ "name": "ghcr.io/soit-ai/soit/server", "digest": {"sha256": "96b80ae141000adde27cf3dedb27935b5f5d0085d69025cffb4bbb63276d5929"} }], "predicateType": "https://slsa.dev/provenance/v1", "predicate": { "buildDefinition": { "buildType": "https://actions.github.io/buildtypes/workflow/v1", "externalParameters": { "workflow": { "ref": "refs/tags/v1.0.0", "repository": "https://github.com/soit-ai/soit", "path": ".github/workflows/release.yml" } }, "internalParameters": { "github": { "event_name": "push", "repository_id": "910429753", "repository_owner_id": "193298865", "runner_environment": "github-hosted" } }, "resolvedDependencies": [{ "uri": "git+https://github.com/soit-ai/soit@refs/tags/v1.0.0", "digest": {"gitCommit": "8105cae074f1f27d7916acfe02f9d4eabb63169f"} }] }, "runDetails": { "builder": {"id": "https://github.com/soit-ai/soit/.github/workflows/release.yml@refs/tags/v1.0.0"}, "metadata": {"invocationId": "https://github.com/soit-ai/soit/actions/runs/31023642816/attempts/1"} } } } ``` One complete sentence: **commit `8105cae…` on `refs/tags/v1.0.0`, built by `.github/workflows/release.yml` on a GitHub-hosted runner in run 31023642816, produced the image with digest `96b80ae1…`.** I had the repository on disk, so I closed the loop immediately: ``` git rev-parse v1.0.0^{commit} # 8105cae074f1f27d7916acfe02f9d4eabb63169f ``` Match. Note `repository_id` and `repository_owner_id`: numeric IDs are worth more than the repository name, because **names can be renamed and transferred, IDs cannot.** `verificationMaterial.certificate.rawBytes` is a 1735-byte DER certificate: ``` issuer=O = sigstore.dev, CN = sigstore-intermediate subject= notBefore=Aug 5 16:14:11 2026 GMT notAfter =Aug 5 16:24:11 2026 GMT X509v3 Subject Alternative Name: critical URI:https://github.com/soit-ai/soit/.github/workflows/release.yml@refs/tags/v1.0.0 ``` Two things worth stopping on. **The subject is empty.** The field that traditionally names the holder is blank; the identity moved entirely into that SAN URI, and it is not a person or an organisation — **it is a workflow at a ref.** **The validity window is exactly ten minutes.** That is keyless signing: the pipeline trades an OIDC token for a short-lived certificate, signs, and throws the key away. Nothing to store, nothing to rotate, nothing to steal later. Ten minutes on, the certificate is expired — and the signature still verifies, for the reason in section 5. The certificate also carries a block of Sigstore extensions describing the build. The values below are what I read; **I am deliberately not putting names on the OIDs** — those live in Fulcio's OID documentation, and the values speak for themselves: | OID suffix | Value | |---|---| | `.1.1` /`.1.8` | `https://token.actions.githubusercontent.com` | | `.1.2` /`.1.20` | `push` | | `.1.3` /`.1.10` /`.1.13` /`.1.19` | `8105cae074f1f27d7916acfe02f9d4eabb63169f` | | `.1.4` | `release` | | `.1.5` | `soit-ai/soit` | | `.1.6` /`.1.14` | `refs/tags/v1.0.0` | | `.1.9` /`.1.18` | `…/release.yml@refs/tags/v1.0.0` | | `.1.11` | `github-hosted` | | `.1.12` | `https://github.com/soit-ai/soit` | | `.1.15` /`.1.17` | `910429753` /`193298865` | | `.1.16` | `https://github.com/soit-ai` | | `.1.21` | `…/actions/runs/31023642816/attempts/1` | | `.1.22` | `public` | | `.1.24` | `repo:soit-ai/soit:ref:refs/tags/v1.0.0` | Extended Key Usage is `Code Signing`, Key Usage is `Digital Signature` only, and there is a CT precertificate SCT timestamped `Aug 5 16:14:11.958 2026 GMT` — the act of issuing this certificate was itself logged publicly. `verificationMaterial.tlogEntries` holds exactly one record: ``` php logIndex : 2346649359 integratedTime : 1785946452 -> 2026-08-05T16:14:12Z kindVersion : {kind: dsse, version: 0.0.1} ``` That record lives in Rekor, the public append-only transparency log, and its job is to witness *when*. The question a verifier asks is not "is this certificate valid now" — it expired long ago — but "**was it valid at the moment of signing**". The log pins that moment at `16:14:12Z`, inside the `16:14:11`–` 16:24:11` window. A detail from the actual bundle: `timestampVerificationData` is an **empty object**. No RFC 3161 timestamp; the Rekor entry carries the time on its own. Not wrong, just worth knowing what your trust actually rests on. The second bundle's predicate is a complete SPDX document. The bundle blob is 3437645 bytes: ``` spdxVersion : SPDX-2.3 dataLicense : CC0-1.0 name : ghcr.io/soit-ai/soit/server creationInfo : creators ["Organization: Anchore, Inc", "Tool: syft-1.42.3"], created "2026-08-05T16:14:08Z" packages : 710 relationships : 2840 ``` Those 710 packages by purl type: **469 `pkg:deb`**, **217 `pkg:pypi`**, one `pkg:generic` (`python 3.11.15`) and one `pkg:oci` (the image itself). The 469 Debian packages come from the base image — `server/Dockerfile` starts at `python:3.11` — and only 217 are our own Python dependencies. That ratio is itself a finding: **two thirds of what you just signed, you did not write.** All three images side by side show the split clearly: | Image | Packages | Composition | SBOM generated | |---|---|---|---| | `server` | 710 | 469 deb + 217 pypi | 16:14:08Z | | `knowledge-worker` | 775 | 469 deb + 281 pypi | 16:22:23Z | | `web` | 1331 | 1311 npm + 18 apk | 16:15:01Z | The 64 extra pypi packages in `knowledge-worker` include `torch`, `transformers` and `nvidia-cublas-cu12`. The two images are two targets of one Dockerfile; the entire difference is `uv sync --extra knowledge-worker` (`server/Dockerfile:19` vs `:27`). `web` is `node:24-alpine`, so the Debian packages become 18 apk ones. An SPDX document normally carries a `files` section recording which files each package was identified from. All three of ours have none — and it is deliberate, right there in the workflow (`.github/workflows/release.yml:132–149`): ``` jq 'del(.files) | .packages |= map(del(.hasFiles)) | .relationships |= map(select( ((.spdxElementId // "") | startswith("SPDXRef-File") | not) and ((.relatedSpdxElement // "") | startswith("SPDXRef-File") | not)))' \ "$f" > "$f.tmp" mv "$f.tmp" "$f" size="$(stat -c%s "$f")" test "$size" -le 16000000 ``` The comment above it gives the reason: per-file SPDX entries push an ML-heavy image's SBOM past the 16 MiB subject limit of `actions/attest`. In plain terms, **the SBOM had to be shrunk until it could be signed.** The cost is real. You keep the package inventory and the package-to-package relationships; you lose "which file did this package come from", which is exactly what you want when a CVE lands and you need to locate it. The `test -le 16000000` line is a hard gate: if a future image blows past the ceiling the release **fails there** rather than shipping an SBOM that cannot be signed. That part I think is right — better a broken build than a half-signed artifact. Two things I found in ours. **Five Windows launchers in a Linux image.** Among the 710 packages, `Simple Launcher 1.1.0.14` appears **five times**, each with a CPE and no purl: ``` SPDXRef-Package-binary-Simple-Launcher-d60858f6579e7bb1 versionInfo : 1.1.0.14 externalRefs: cpe:2.3:a:Simple_Launcher:Simple_Launcher:1.1.0.14:*:*:*:*:*:*:* ``` Syft's binary classifier found them: Python packaging wheels ship small Windows launcher executables, which will never execute inside this Linux image but are inventoried all the same. And because the `files` section was stripped (section 7), you cannot even find out where they are. **A fourth hash that matches nothing.** The `pkg:oci` entry representing the image itself reads: ``` pkg:oci/ghcr.io%2Fsoit-ai%2Fsoit%2Fserver@sha256%3Ae7c993b9ac5d7322058e0169c678b4ce21fe42fac72c5d3374404bf4642939ea?arch=amd64 ``` `e7c993b9…` is **not** the index digest, **not** the platform manifest digest and **not** the config digest. Asked for directly, the registry returns 404. I could not determine what it is — the reasonable guess is an identifier the scanner computes locally — so I am reporting the observation and not a conclusion. The practical rule is clear enough: **do not reconcile an SBOM's internal digest against the signature's subject.** The binding lives in exactly one place, `subject.digest`, and that value is` 96b80ae1…`, matching the registry exactly.` v1.0.0` images, all of them verifiable Back to those two extra tags. Fetched the same way: | Image digest behind the fallback tag | Attestations attached | Signed at | |---|---|---| | `3a5b3b1a…8246` | provenance only | 2026-08-05T15:41:15Z | | `236201a5…f874f` | provenance + SBOM | 15:52:32Z / 15:52:44Z | | `96b80ae1…5929` | provenance + SBOM | 16:14:12Z / 16:14:22Z | All three provenance statements name `refs/tags/v1.0.0` and the same signing identity. **The commits differ:** | Image digest | Commit in provenance | Actions run | Commit subject | |---|---|---|---| | `3a5b3b1a…` | `0dacfc52…` | 31020921135 | `ci(release): create the artifacts directory before image SBOM generation` | | `236201a5…` | `ec822c63…` | 31021873557 | `ci(release): catalog packages only in image SBOMs` | | `96b80ae1…` | `8105cae0…` | 31023642816 | `ci(release): trim image SBOMs to package level before attestation` | All three commits are on `main` locally, 23:33 / 23:44 / 00:05 in my timezone. The story reads itself off the commit subjects: **the tag was re-pointed twice that night** while the SBOM step was being fixed — and each failed attempt had already So this happens: ``` gh attestation verify \ oci://ghcr.io/soit-ai/soit/server@sha256:3a5b3b1a...8246 \ --repo soit-ai/soit --format json exit code : 0 subject : 3a5b3b1a...8246 predicate : https://slsa.dev/provenance/v1 ref : refs/tags/v1.0.0 commit : 0dacfc5219c4ae57d024f9cb4208e1efad5f0d17 san : https://github.com/soit-ai/soit/.github/workflows/release.yml@refs/tags/v1.0.0 tlog : rekor.sigstore.dev @ 2026-08-05T23:41:11+08:00 ``` **Exit 0.** Nothing is wrong with the signature. It states honestly what it is: an image built by our release workflow, on `refs/tags/v1.0.0`, from commit `0dacfc52…`. **The one thing it does not say is that it is not the image we shipped.** No attack here — we built all three. But swap in a nastier scenario and it holds immediately: a pipeline compromised once, fixed, re-tagged, with the poisoned intermediate still sitting in the registry. **Attestation will not catch that.** `gh attestation verify` answers *"did this come out of that pipeline"*. It never promised to answer *"is this the release"*. That answer is not in the signature. It is an asset on the GitHub Release, which carries seven files: ``` soit-1.0.0.tar.gz 3148831 SHA256SUMS 433 release-artifacts.json 2877 server.spdx.json 3313874 knowledge-worker.spdx.json 4004516 web.spdx.json 3862614 source.spdx.json 3694033 ``` `release-artifacts.json` is the manifest: ``` { "featureKey": "release.artifacts", "schemaVersion": 1, "version": "1.0.0", "release_tag": "v1.0.0", "commit": "8105cae074f1f27d7916acfe02f9d4eabb63169f", "clean_worktree_at_tag": true, "images": [{ "component": "server", "name": "ghcr.io/soit-ai/soit/server", "release_tag": "v1.0.0", "digest": "sha256:96b80ae141000adde27cf3dedb27935b5f5d0085d69025cffb4bbb63276d5929", "reference": "ghcr.io/soit-ai/soit/server@sha256:96b80ae1...5929", "sbom": {"format": "spdx-json", "path": "server.spdx.json", "sha256": "b83c3bc7...ef68", "attestation": "..."}, "provenance_attestation": "..." }] } ``` It says the thing the signature would not: **`v1.0.0` is `96b80ae1…`, not the other two.** It is assembled by `jq` inside the `publish-release` job (`release.yml:300–347`) and validated on the spot. I fed the copy downloaded from the public internet straight into the same script that lives in the repository: ``` { "commit": "8105cae074f1f27d7916acfe02f9d4eabb63169f", "images": ["knowledge-worker", "server", "web"], "passed": true, "release_tag": "v1.0.0" } exit=0 ``` 178 lines, standard library only, no network (`server/scripts/verify_release_artifacts.py`). It insists that: `release_tag` equals `v` + `version` (`:42`);` commit` is 40 lowercase hex characters and not all zeroes (`:45`);` clean_worktree_at_tag` is `true` (`:49`);` reference` `name@digest``:91`) — a `name:tag` reference is rejected outright;`digest` matches `^sha256:[0-9a-f]{64}$` (`:88`);`:15`, `:114`);`:107–112`). A test guards that last-but-two rule: it rewrites the example's `reference` into tag form and asserts the script fails with `digest-pinned` (`server/tests/unit/test_release_operations_contract.py:85–88`). The release also ships a `git archive` tarball (`release.yml:253–263`) described as deterministic. Worth testing rather than believing. First, the checksum: ``` grep "soit-1.0.0.tar.gz" SHA256SUMS | sha256sum -c - # ./soit-1.0.0.tar.gz: OK ``` Then rebuild it locally: ``` git archive --format=tar.gz --prefix="soit-1.0.0/" \ --output=local.tar.gz 8105cae074f1f27d7916acfe02f9d4eabb63169f c11179c379ba7390c215133f342c92a7517a548f1536959cb43e187b16a7bab3 *local.tar.gz 5d3dd50f491a897ba9a184ad6dc474e2ef55508224404266d91b3e1e1d32ae9d *soit-1.0.0.tar.gz ``` Different — and not just at the gzip layer; the inner tars differ too. Comparing members explains it: ``` members : 1551 local / 1551 released -- same only-one-side: 0 / 0 -- same differing : 1280 members mtime : 1785945959 everywhere -- same mode : 0664 everywhere -- same example: soit-1.0.0/.github/workflows/quality.yml local 15270 released 14803 ``` Sizes only, and each delta equals that file's line count. **CRLF.** My machine has `core.autocrlf=true`, so `git archive` helpfully rewrote the line endings. Turn it off: ``` git -c core.autocrlf=false archive --format=tar.gz --prefix="soit-1.0.0/" \ --output=local2.tar.gz 8105cae074f1f27d7916acfe02f9d4eabb63169f # 5d3dd50f491a897ba9a184ad6dc474e2ef55508224404266d91b3e1e1d32ae9d *local2.tar.gz ``` **Identical.** A Windows box, git 2.55.0, a month later, reproducing the tarball GitHub Actions produced. Both halves matter. Determinism is real — `git archive` writes no build timestamp and takes mtimes from the commit — **but it is sensitive to local git configuration**, and the first suspect when reproduction fails is `core.autocrlf`, not the publisher. Everything above, compressed into something you can paste: ``` # 1. Resolve the digest (never keep books against a tag) docker buildx imagetools inspect ghcr.io/soit-ai/soit/server:v1.0.0 | head -3 # 2. Verify build provenance gh attestation verify oci://ghcr.io/soit-ai/soit/server:v1.0.0 --repo soit-ai/soit # 3. Verify the SBOM attestation (NOT checked by default) gh attestation verify oci://ghcr.io/soit-ai/soit/server:v1.0.0 --repo soit-ai/soit \ --predicate-type https://spdx.dev/Document/v2.3 # 4. Check the release manifest: is this digest the one that shipped curl -sLO https://github.com/soit-ai/soit/releases/download/v1.0.0/release-artifacts.json curl -sLO https://github.com/soit-ai/soit/releases/download/v1.0.0/SHA256SUMS sha256sum -c SHA256SUMS # 5. Run it by digest, not by tag docker pull ghcr.io/soit-ai/soit/server@sha256:96b80ae1...5929 ``` Step 3 deserves emphasis: **`gh attestation verify` checks provenance only by default.** Without `--predicate-type` it returned exactly one result here, the SLSA provenance. With it, the SPDX attestation comes back — 6022876 bytes of JSON holding the same 710 packages counted in section 6. A lot of people assume "verified" includes the SBOM. It does not. Step 4 is the point of this whole piece. Without it, the green checkmark from step 2 shines just as brightly on the `3a5b3b1a…` image from section 9. House rule: the last section is about our own problems. **1. Our own compose pins tags, not digests.** All six services in `docker/docker-compose.images.yml` use `:${SOIT_IMAGE_TAG:-v1.0.0}` (`:17`–`:32`), and the README says the same (`README.md:146`). We demand digest-pinned references inside `release-artifacts.json` and then hand users a tag. Impact: a tag can be re-pointed. Workaround: step 5 of the checklist. **I plan to open an issue**; none filed at the time of writing, hence no link. **2. `SHA256SUMS` does not cover `release-artifacts.json`.** The 433-byte file lists five entries — four SPDX documents and the tarball. Neither the manifest nor `SHA256SUMS` itself is in there (the latter cannot be, by construction). The manifest does get its own attestation (`release.yml:350–353`), so it is not unprotected, but **the instinctive "download everything and run `sha256sum -c`" does not reach it.** **Also going to open an issue.** **3. `docs/release-process.md` lags the pipeline.** It tells you to *copy* the real tag, commit, digests, SBOM checksums and attestation URLs into an evidence document and then run the verifier (`:24`–`:32`). The pipeline has been doing that automatically for a while, and uploads the result as a release asset. Impact: a reader concludes the manifest is hand-written after the fact, which makes it look weaker than it is. **Also going to open an issue.** **4. Superseded builds are never cleaned up.** The two images from section 9 are still pullable, still verifiable, still runnable. We have no "delete same-tag intermediates after a successful release" step. Impact: as described. Workaround: pull the digest from `release-artifacts.json`. **5. The verifier checks shape, not truth.** `provenance_attestation` only has to be a non-empty string (`:58`, `:64`, `:106`, all via `_require_text`). The script never fetches the URL and never compares the attested subject digest to the one in the manifest. Impact: a well-formed manifest full of fabricated URLs passes. Workaround: it is a **structural** check, not a **trust** check — the trust comes from `gh attestation verify`, and you need both. Defensible division of labour, but the docs never say so. **6. `REQUIRED_IMAGES` is hard-coded.** `verify_release_artifacts.py:15` pins `{"server", "knowledge-worker", "web"}` and line 114 demands exact equality. Add a fourth image and the release breaks at the final step. Impact: a bad surprise on release day. Workaround: know it is there — and the case it blocks (silently shipping with an image missing) is worth more than the nuisance. **7. No process for SBOM false positives.** Sections 7 and 8. Nobody owns annotating `Simple Launcher` as noise, so the next reader has to work it out again. Impact: the signal-to-noise ratio degrades as images grow. **8. I could not identify `e7c993b9…`.** That is where section 8 stops. **This entry exists so I do not pretend otherwise**: I can confirm it is absent from the registry `8105cae…`). Later releases will have different values; the method does not change.`gh attestation verify` **A passing attestation says "these bytes came out of that pipeline"; it does not say "this is the release."** What joins the two is a manifest binding tag, commit and `release-artifacts.json`, watched by a 178-line script whose hardest The repository is `github.com/soit-ai/soit`, and every step above is reproducible: `curl` calls in section 2 need no account — an anonymous token is enough;`3a5b3b1a…` image from section 9 is still there; verify it yourself and watch it exit 0;`-c core.autocrlf=false`. If any claim here is wrong — especially the digest in section 8 that I failed to identify — open an issue and say so. I would rather learn where this is wrong than be told the pipeline looks thorough.