{"slug": "gh-attestation-verify-said-yes-to-an-image-we-never-released", "title": "`gh attestation verify` said yes to an image we never released", "summary": "A developer demonstrated a four-step container image verification workflow against the soit-ai/soit server v1.0.0 release, showing that fetching by digest, verifying provenance, reading the SBOM, and checking the release manifest are all required to confirm an image's authenticity. The walkthrough highlights that the OCI referrers API returned MANIFEST_UNKNOWN for both the index and platform manifest digests, and that the spec's tag-based fallback (sha256-<digest>) is needed to locate the Sigstore provenance and SPDX SBOM attestations. The author warns that skipping the release manifest check means verifying a different build than the one a version actually shipped.", "body_md": "Pulling a container image somebody else built asks four separate questions. Most\n\npeople answer three of them and skip the fourth:\n\n| Step | Question it answers | If you skip it | \n|---|---|---|\n| Fetch by digest | are these the bytes I asked for | you are gambling on a mutable tag | \n| Verify provenance | who built these bytes, from what, where | unknown origin | \n| Read the SBOM | what is inside | a CVE lands and you cannot tell if you are affected | \n| Check the release manifest | **is this digest the one that version shipped** | you may be verifying a different build | \n\nThe fourth is the one that has no substitute, and it is the one that is almost\n\nalways skipped. What follows is all four, run for real against our own `v1.0.0`.\n\nThis is where the confusion starts. Get an anonymous pull token — a public package\n\nneeds no account:\n\n```\nTOKEN=$(curl -s \"https://ghcr.io/token?scope=repository:soit-ai/soit/server:pull&service=ghcr.io\" \\\n  | sed -n 's/.*\"token\":\"\\([^\"]*\\)\".*/\\1/p')\n```\n\nAsk what the `v1.0.0` tag resolves to:\n\n```\ncurl -sI -H \"Authorization: Bearer $TOKEN\" \\\n  -H \"Accept: application/vnd.oci.image.index.v1+json\" \\\n  https://ghcr.io/v2/soit-ai/soit/server/manifests/v1.0.0\ndocker-content-digest: sha256:96b80ae141000adde27cf3dedb27935b5f5d0085d69025cffb4bbb63276d5929\nContent-Type: application/vnd.oci.image.index.v1+json\nContent-Length: 856\n```\n\n856 bytes: this is an image index, not the image. Inside it, two entries:\n\n```\nmanifests[0]: sha256:84e7f539...d32f  2589 bytes  linux/amd64\nmanifests[1]: sha256:a6bd430d...e1be   565 bytes  unknown/unknown\n              vnd.docker.reference.type:   attestation-manifest\n              vnd.docker.reference.digest: sha256:84e7f539...d32f\n```\n\nThat `unknown/unknown` entry is not corruption — it is BuildKit's own attestation\n\nfor the amd64 manifest, and it is **a different mechanism from the Sigstore signatures** we are about to read. First thing people conflate.\n\nOne level down, the amd64 manifest says:\n\n```\nconfig: sha256:021814982fcc214a02a2b322cbc3729b3ce43bd6e191aa0fa39da540a3e3cd1f  8769 bytes\nlayers: 12, 684216450 bytes compressed (about 652 MiB)\n```\n\nThree hashes so far, three different meanings:\n\n| Name | Value for server v1.0.0 | Refers to | \n|---|---|---|\n| index digest | `96b80ae1…5929` | the multi-platform index itself | \n| platform manifest digest | `84e7f539…d32f` | the linux/amd64 image | \n| config digest | `02181498…cd1f` | the image config JSON | \n\n**The signature covers the first one.** A fourth hash shows up in section 8, and it\n\nis none of these.\n\nThe OCI spec defines a referrers API for \"things attached to this thing\":\n\n```\ncurl -s -H \"Authorization: Bearer $TOKEN\" \\\n  https://ghcr.io/v2/soit-ai/soit/server/referrers/sha256:96b80ae1...5929\n{\"errors\":[{\"code\":\"MANIFEST_UNKNOWN\",\"message\":\"manifest unknown\"}]}\n```\n\nSame error for the platform manifest digest. **Do not conclude \"unsigned\" from this.** The spec has a fallback: replace the colon with a dash and treat the digest\n\n```\n{\"name\":\"soit-ai/soit/server\",\"tags\":[\n  \"v1.0.0\",\n  \"sha256-3a5b3b1a2d14e0646298826602124ba22e63f8c078f653e506a0a042bfd18246\",\n  \"sha256-236201a5a3ee861ffdb5fdd7ec134454619eb1ab0e777439c4a22a65002f874f\",\n  \"sha256-96b80ae141000adde27cf3dedb27935b5f5d0085d69025cffb4bbb63276d5929\"]}\n```\n\nThe last one is ours. **Remember the first two — section 9 is entirely about them.**\n\nFetching `sha256-96b80ae1…` returns a small index holding two Sigstore bundles:\n\n```\nsha256:658b3428...80b9   812 bytes\n  artifactType : application/vnd.dev.sigstore.bundle.v0.3+json\n  predicateType: https://slsa.dev/provenance/v1\n  created      : 2026-08-05T16:14:12.946Z\n\nsha256:59bac8c5...d9cf   814 bytes\n  predicateType: https://spdx.dev/Document/v2.3\n  created      : 2026-08-05T16:14:22.004Z\n```\n\nBuild provenance and an SBOM attestation, signed nine seconds apart. Both manifests\n\ncarry `subject` pointing back at `sha256:96b80ae1…5929` — **the single place where \"this signature belongs to that image\" is actually written down.**\n\nPull the bundle blob and hash it:\n\n```\ncurl -sL -H \"Authorization: Bearer $TOKEN\" \\\n  https://ghcr.io/v2/soit-ai/soit/server/blobs/sha256:eae5d902...dd86 -o bundle.json\nsha256sum bundle.json\n# eae5d902186b490570aba6f702f7ce03c46f935e403bf881ec27d99480d7dd86 *bundle.json\n```\n\nThe hash equals the digest I asked for. A content-addressed registry means this step\n\nverifies itself; no trust required yet.\n\nThe bundle has three top-level fields — `mediaType`, `verificationMaterial`,\n\n`dsseEnvelope`. The envelope's `payloadType` is `application/vnd.in-toto+json`, and\n\nthe base64 payload is the only thing the signature covers:\n\n```\n{\n  \"_type\": \"https://in-toto.io/Statement/v1\",\n  \"subject\": [{\n    \"name\": \"ghcr.io/soit-ai/soit/server\",\n    \"digest\": {\"sha256\": \"96b80ae141000adde27cf3dedb27935b5f5d0085d69025cffb4bbb63276d5929\"}\n  }],\n  \"predicateType\": \"https://slsa.dev/provenance/v1\",\n  \"predicate\": {\n    \"buildDefinition\": {\n      \"buildType\": \"https://actions.github.io/buildtypes/workflow/v1\",\n      \"externalParameters\": {\n        \"workflow\": {\n          \"ref\": \"refs/tags/v1.0.0\",\n          \"repository\": \"https://github.com/soit-ai/soit\",\n          \"path\": \".github/workflows/release.yml\"\n        }\n      },\n      \"internalParameters\": {\n        \"github\": {\n          \"event_name\": \"push\",\n          \"repository_id\": \"910429753\",\n          \"repository_owner_id\": \"193298865\",\n          \"runner_environment\": \"github-hosted\"\n        }\n      },\n      \"resolvedDependencies\": [{\n        \"uri\": \"git+https://github.com/soit-ai/soit@refs/tags/v1.0.0\",\n        \"digest\": {\"gitCommit\": \"8105cae074f1f27d7916acfe02f9d4eabb63169f\"}\n      }]\n    },\n    \"runDetails\": {\n      \"builder\": {\"id\": \"https://github.com/soit-ai/soit/.github/workflows/release.yml@refs/tags/v1.0.0\"},\n      \"metadata\": {\"invocationId\": \"https://github.com/soit-ai/soit/actions/runs/31023642816/attempts/1\"}\n    }\n  }\n}\n```\n\nOne complete sentence: **commit `8105cae…` on `refs/tags/v1.0.0`, built by\n`.github/workflows/release.yml` on a GitHub-hosted runner in run 31023642816,\nproduced the image with digest `96b80ae1…`.**\n\nI had the repository on disk, so I closed the loop immediately:\n\n```\ngit rev-parse v1.0.0^{commit}\n# 8105cae074f1f27d7916acfe02f9d4eabb63169f\n```\n\nMatch. Note `repository_id` and `repository_owner_id`: numeric IDs are worth more\n\nthan the repository name, because **names can be renamed and transferred, IDs cannot.**\n\n`verificationMaterial.certificate.rawBytes` is a 1735-byte DER certificate:\n\n```\nissuer=O = sigstore.dev, CN = sigstore-intermediate\nsubject=\nnotBefore=Aug  5 16:14:11 2026 GMT\nnotAfter =Aug  5 16:24:11 2026 GMT\nX509v3 Subject Alternative Name: critical\n    URI:https://github.com/soit-ai/soit/.github/workflows/release.yml@refs/tags/v1.0.0\n```\n\nTwo things worth stopping on.\n\n**The subject is empty.** The field that traditionally names the holder is blank; the\n\nidentity moved entirely into that SAN URI, and it is not a person or an organisation —\n\n**it is a workflow at a ref.**\n\n**The validity window is exactly ten minutes.** That is keyless signing: the pipeline\n\ntrades an OIDC token for a short-lived certificate, signs, and throws the key away.\n\nNothing to store, nothing to rotate, nothing to steal later. Ten minutes on, the\n\ncertificate is expired — and the signature still verifies, for the reason in section 5.\n\nThe certificate also carries a block of Sigstore extensions describing the build. The\n\nvalues below are what I read; **I am deliberately not putting names on the OIDs** —\n\nthose live in Fulcio's OID documentation, and the values speak for themselves:\n\n| OID suffix | Value | \n|---|---|\n| `.1.1` /`.1.8` | `https://token.actions.githubusercontent.com` | \n| `.1.2` /`.1.20` | `push` | \n| `.1.3` /`.1.10` /`.1.13` /`.1.19` | `8105cae074f1f27d7916acfe02f9d4eabb63169f` | \n| `.1.4` | `release` | \n| `.1.5` | `soit-ai/soit` | \n| `.1.6` /`.1.14` | `refs/tags/v1.0.0` | \n| `.1.9` /`.1.18` | `…/release.yml@refs/tags/v1.0.0` | \n| `.1.11` | `github-hosted` | \n| `.1.12` | `https://github.com/soit-ai/soit` | \n| `.1.15` /`.1.17` | `910429753` /`193298865` | \n| `.1.16` | `https://github.com/soit-ai` | \n| `.1.21` | `…/actions/runs/31023642816/attempts/1` | \n| `.1.22` | `public` | \n| `.1.24` | `repo:soit-ai/soit:ref:refs/tags/v1.0.0` | \n\nExtended Key Usage is `Code Signing`, Key Usage is `Digital Signature` only, and\n\nthere is a CT precertificate SCT timestamped `Aug 5 16:14:11.958 2026 GMT` — the act\n\nof issuing this certificate was itself logged publicly.\n\n`verificationMaterial.tlogEntries` holds exactly one record:\n\n``` php\nlogIndex        : 2346649359\nintegratedTime  : 1785946452   ->  2026-08-05T16:14:12Z\nkindVersion     : {kind: dsse, version: 0.0.1}\n```\n\nThat record lives in Rekor, the public append-only transparency log, and its job is\n\nto witness *when*. The question a verifier asks is not \"is this certificate valid\n\nnow\" — it expired long ago — but \"**was it valid at the moment of signing**\". The log\n\npins that moment at `16:14:12Z`, inside the `16:14:11`–` 16:24:11` window.\n\nA detail from the actual bundle: `timestampVerificationData` is an **empty object**.\n\nNo RFC 3161 timestamp; the Rekor entry carries the time on its own. Not wrong, just\n\nworth knowing what your trust actually rests on.\n\nThe second bundle's predicate is a complete SPDX document. The bundle blob is\n\n3437645 bytes:\n\n```\nspdxVersion   : SPDX-2.3\ndataLicense   : CC0-1.0\nname          : ghcr.io/soit-ai/soit/server\ncreationInfo  : creators [\"Organization: Anchore, Inc\", \"Tool: syft-1.42.3\"],\n                created \"2026-08-05T16:14:08Z\"\npackages      : 710\nrelationships : 2840\n```\n\nThose 710 packages by purl type: **469 `pkg:deb`**, **217 `pkg:pypi`**, one\n\n`pkg:generic` (`python 3.11.15`) and one `pkg:oci` (the image itself).\n\nThe 469 Debian packages come from the base image — `server/Dockerfile` starts at\n\n`python:3.11` — and only 217 are our own Python dependencies. That ratio is itself a\n\nfinding: **two thirds of what you just signed, you did not write.**\n\nAll three images side by side show the split clearly:\n\n| Image | Packages | Composition | SBOM generated | \n|---|---|---|---|\n| `server` | 710 | 469 deb + 217 pypi | 16:14:08Z | \n| `knowledge-worker` | 775 | 469 deb + 281 pypi | 16:22:23Z | \n| `web` | 1331 | 1311 npm + 18 apk | 16:15:01Z | \n\nThe 64 extra pypi packages in `knowledge-worker` include `torch`, `transformers` and\n\n`nvidia-cublas-cu12`. The two images are two targets of one Dockerfile; the entire\n\ndifference is `uv sync --extra knowledge-worker` (`server/Dockerfile:19` vs `:27`).\n\n`web` is `node:24-alpine`, so the Debian packages become 18 apk ones.\n\nAn SPDX document normally carries a `files` section recording which files each\n\npackage was identified from. All three of ours have none — and it is deliberate,\n\nright there in the workflow (`.github/workflows/release.yml:132–149`):\n\n```\njq 'del(.files)\n    | .packages |= map(del(.hasFiles))\n    | .relationships |= map(select(\n        ((.spdxElementId // \"\") | startswith(\"SPDXRef-File\") | not)\n        and ((.relatedSpdxElement // \"\") | startswith(\"SPDXRef-File\") | not)))' \\\n  \"$f\" > \"$f.tmp\"\nmv \"$f.tmp\" \"$f\"\nsize=\"$(stat -c%s \"$f\")\"\ntest \"$size\" -le 16000000\n```\n\nThe comment above it gives the reason: per-file SPDX entries push an ML-heavy image's\n\nSBOM past the 16 MiB subject limit of `actions/attest`. In plain terms, **the SBOM had to be shrunk until it could be signed.**\n\nThe cost is real. You keep the package inventory and the package-to-package\n\nrelationships; you lose \"which file did this package come from\", which is exactly what\n\nyou want when a CVE lands and you need to locate it. The `test -le 16000000` line is a\n\nhard gate: if a future image blows past the ceiling the release **fails there** rather\n\nthan shipping an SBOM that cannot be signed. That part I think is right — better a\n\nbroken build than a half-signed artifact.\n\nTwo things I found in ours.\n\n**Five Windows launchers in a Linux image.** Among the 710 packages,\n\n`Simple Launcher 1.1.0.14` appears **five times**, each with a CPE and no purl:\n\n```\nSPDXRef-Package-binary-Simple-Launcher-d60858f6579e7bb1\n  versionInfo : 1.1.0.14\n  externalRefs: cpe:2.3:a:Simple_Launcher:Simple_Launcher:1.1.0.14:*:*:*:*:*:*:*\n```\n\nSyft's binary classifier found them: Python packaging wheels ship small Windows\n\nlauncher executables, which will never execute inside this Linux image but are\n\ninventoried all the same. And because the `files` section was stripped (section 7),\n\nyou cannot even find out where they are.\n\n**A fourth hash that matches nothing.** The `pkg:oci` entry representing the image\n\nitself reads:\n\n```\npkg:oci/ghcr.io%2Fsoit-ai%2Fsoit%2Fserver@sha256%3Ae7c993b9ac5d7322058e0169c678b4ce21fe42fac72c5d3374404bf4642939ea?arch=amd64\n```\n\n`e7c993b9…` is **not** the index digest, **not** the platform manifest digest and\n\n**not** the config digest. Asked for directly, the registry returns 404. I could not\n\ndetermine what it is — the reasonable guess is an identifier the scanner computes\n\nlocally — so I am reporting the observation and not a conclusion. The practical rule\n\nis clear enough: **do not reconcile an SBOM's internal digest against the signature's subject.** The binding lives in exactly one place, \n\n`subject.digest`, and that value is` 96b80ae1…`, matching the registry exactly.` v1.0.0` images, all of them verifiable\nBack to those two extra tags. Fetched the same way:\n\n| Image digest behind the fallback tag | Attestations attached | Signed at | \n|---|---|---|\n| `3a5b3b1a…8246` | provenance only | 2026-08-05T15:41:15Z | \n| `236201a5…f874f` | provenance + SBOM | 15:52:32Z / 15:52:44Z | \n| `96b80ae1…5929` | provenance + SBOM | 16:14:12Z / 16:14:22Z | \n\nAll three provenance statements name `refs/tags/v1.0.0` and the same signing identity.\n\n**The commits differ:**\n\n| Image digest | Commit in provenance | Actions run | Commit subject | \n|---|---|---|---|\n| `3a5b3b1a…` | `0dacfc52…` | 31020921135 | `ci(release): create the artifacts directory before image SBOM generation` | \n| `236201a5…` | `ec822c63…` | 31021873557 | `ci(release): catalog packages only in image SBOMs` | \n| `96b80ae1…` | `8105cae0…` | 31023642816 | `ci(release): trim image SBOMs to package level before attestation` | \n\nAll three commits are on `main` locally, 23:33 / 23:44 / 00:05 in my timezone. The\n\nstory reads itself off the commit subjects: **the tag was re-pointed twice that night** while the SBOM step was being fixed — and each failed attempt had already\n\nSo this happens:\n\n```\ngh attestation verify \\\n  oci://ghcr.io/soit-ai/soit/server@sha256:3a5b3b1a...8246 \\\n  --repo soit-ai/soit --format json\nexit code : 0\nsubject   : 3a5b3b1a...8246\npredicate : https://slsa.dev/provenance/v1\nref       : refs/tags/v1.0.0\ncommit    : 0dacfc5219c4ae57d024f9cb4208e1efad5f0d17\nsan       : https://github.com/soit-ai/soit/.github/workflows/release.yml@refs/tags/v1.0.0\ntlog      : rekor.sigstore.dev @ 2026-08-05T23:41:11+08:00\n```\n\n**Exit 0.** Nothing is wrong with the signature. It states honestly what it is: an\n\nimage built by our release workflow, on `refs/tags/v1.0.0`, from commit `0dacfc52…`.\n\n**The one thing it does not say is that it is not the image we shipped.**\n\nNo attack here — we built all three. But swap in a nastier scenario and it holds\n\nimmediately: a pipeline compromised once, fixed, re-tagged, with the poisoned\n\nintermediate still sitting in the registry. **Attestation will not catch that.**\n\n`gh attestation verify` answers *\"did this come out of that pipeline\"*. It never\n\npromised to answer *\"is this the release\"*.\n\nThat answer is not in the signature. It is an asset on the GitHub Release, which\n\ncarries seven files:\n\n```\nsoit-1.0.0.tar.gz          3148831\nSHA256SUMS                     433\nrelease-artifacts.json        2877\nserver.spdx.json           3313874\nknowledge-worker.spdx.json 4004516\nweb.spdx.json              3862614\nsource.spdx.json           3694033\n```\n\n`release-artifacts.json` is the manifest:\n\n```\n{\n  \"featureKey\": \"release.artifacts\",\n  \"schemaVersion\": 1,\n  \"version\": \"1.0.0\",\n  \"release_tag\": \"v1.0.0\",\n  \"commit\": \"8105cae074f1f27d7916acfe02f9d4eabb63169f\",\n  \"clean_worktree_at_tag\": true,\n  \"images\": [{\n    \"component\": \"server\",\n    \"name\": \"ghcr.io/soit-ai/soit/server\",\n    \"release_tag\": \"v1.0.0\",\n    \"digest\": \"sha256:96b80ae141000adde27cf3dedb27935b5f5d0085d69025cffb4bbb63276d5929\",\n    \"reference\": \"ghcr.io/soit-ai/soit/server@sha256:96b80ae1...5929\",\n    \"sbom\": {\"format\": \"spdx-json\", \"path\": \"server.spdx.json\", \"sha256\": \"b83c3bc7...ef68\", \"attestation\": \"...\"},\n    \"provenance_attestation\": \"...\"\n  }]\n}\n```\n\nIt says the thing the signature would not: **`v1.0.0` is `96b80ae1…`, not the other\ntwo.** It is assembled by `jq` inside the `publish-release` job\n\n(`release.yml:300–347`) and validated on the spot. I fed the copy downloaded from the\n\npublic internet straight into the same script that lives in the repository:\n\n```\n{\n  \"commit\": \"8105cae074f1f27d7916acfe02f9d4eabb63169f\",\n  \"images\": [\"knowledge-worker\", \"server\", \"web\"],\n  \"passed\": true,\n  \"release_tag\": \"v1.0.0\"\n}\nexit=0\n```\n\n178 lines, standard library only, no network\n\n(`server/scripts/verify_release_artifacts.py`). It insists that:\n\n`release_tag` equals `v` + `version` (`:42`);` commit` is 40 lowercase hex characters and not all zeroes (`:45`);` clean_worktree_at_tag` is `true` (`:49`);` reference` `name@digest``:91`) — a `name:tag` reference is\nrejected outright;`digest` matches `^sha256:[0-9a-f]{64}$` (`:88`);`:15`, `:114`);`:107–112`).\nA test guards that last-but-two rule: it rewrites the example's `reference` into tag\n\nform and asserts the script fails with `digest-pinned`\n\n(`server/tests/unit/test_release_operations_contract.py:85–88`).\n\nThe release also ships a `git archive` tarball (`release.yml:253–263`) described as\n\ndeterministic. Worth testing rather than believing. First, the checksum:\n\n```\ngrep \"soit-1.0.0.tar.gz\" SHA256SUMS | sha256sum -c -\n# ./soit-1.0.0.tar.gz: OK\n```\n\nThen rebuild it locally:\n\n```\ngit archive --format=tar.gz --prefix=\"soit-1.0.0/\" \\\n  --output=local.tar.gz 8105cae074f1f27d7916acfe02f9d4eabb63169f\nc11179c379ba7390c215133f342c92a7517a548f1536959cb43e187b16a7bab3 *local.tar.gz\n5d3dd50f491a897ba9a184ad6dc474e2ef55508224404266d91b3e1e1d32ae9d *soit-1.0.0.tar.gz\n```\n\nDifferent — and not just at the gzip layer; the inner tars differ too. Comparing\n\nmembers explains it:\n\n```\nmembers   : 1551 local / 1551 released   -- same\nonly-one-side: 0 / 0                     -- same\ndiffering : 1280 members\nmtime     : 1785945959 everywhere        -- same\nmode      : 0664 everywhere              -- same\nexample: soit-1.0.0/.github/workflows/quality.yml   local 15270  released 14803\n```\n\nSizes only, and each delta equals that file's line count. **CRLF.** My machine has\n\n`core.autocrlf=true`, so `git archive` helpfully rewrote the line endings. Turn it\n\noff:\n\n```\ngit -c core.autocrlf=false archive --format=tar.gz --prefix=\"soit-1.0.0/\" \\\n  --output=local2.tar.gz 8105cae074f1f27d7916acfe02f9d4eabb63169f\n# 5d3dd50f491a897ba9a184ad6dc474e2ef55508224404266d91b3e1e1d32ae9d *local2.tar.gz\n```\n\n**Identical.** A Windows box, git 2.55.0, a month later, reproducing the tarball\n\nGitHub Actions produced.\n\nBoth halves matter. Determinism is real — `git archive` writes no build timestamp and\n\ntakes mtimes from the commit — **but it is sensitive to local git configuration**, and\n\nthe first suspect when reproduction fails is `core.autocrlf`, not the publisher.\n\nEverything above, compressed into something you can paste:\n\n```\n# 1. Resolve the digest (never keep books against a tag)\ndocker buildx imagetools inspect ghcr.io/soit-ai/soit/server:v1.0.0 | head -3\n\n# 2. Verify build provenance\ngh attestation verify oci://ghcr.io/soit-ai/soit/server:v1.0.0 --repo soit-ai/soit\n\n# 3. Verify the SBOM attestation (NOT checked by default)\ngh attestation verify oci://ghcr.io/soit-ai/soit/server:v1.0.0 --repo soit-ai/soit \\\n  --predicate-type https://spdx.dev/Document/v2.3\n\n# 4. Check the release manifest: is this digest the one that shipped\ncurl -sLO https://github.com/soit-ai/soit/releases/download/v1.0.0/release-artifacts.json\ncurl -sLO https://github.com/soit-ai/soit/releases/download/v1.0.0/SHA256SUMS\nsha256sum -c SHA256SUMS\n\n# 5. Run it by digest, not by tag\ndocker pull ghcr.io/soit-ai/soit/server@sha256:96b80ae1...5929\n```\n\nStep 3 deserves emphasis: **`gh attestation verify` checks provenance only by\ndefault.** Without `--predicate-type` it returned exactly one result here, the SLSA\n\nprovenance. With it, the SPDX attestation comes back — 6022876 bytes of JSON holding\n\nthe same 710 packages counted in section 6. A lot of people assume \"verified\" includes\n\nthe SBOM. It does not.\n\nStep 4 is the point of this whole piece. Without it, the green checkmark from step 2\n\nshines just as brightly on the `3a5b3b1a…` image from section 9.\n\nHouse rule: the last section is about our own problems.\n\n**1. Our own compose pins tags, not digests.** All six services in\n\n`docker/docker-compose.images.yml` use `:${SOIT_IMAGE_TAG:-v1.0.0}` (`:17`–`:32`), and\n\nthe README says the same (`README.md:146`). We demand digest-pinned references inside\n\n`release-artifacts.json` and then hand users a tag. Impact: a tag can be re-pointed.\n\nWorkaround: step 5 of the checklist. **I plan to open an issue**; none filed at the\n\ntime of writing, hence no link.\n\n**2. `SHA256SUMS` does not cover `release-artifacts.json`.** The 433-byte file lists\n\nfive entries — four SPDX documents and the tarball. Neither the manifest nor\n\n`SHA256SUMS` itself is in there (the latter cannot be, by construction). The manifest\n\ndoes get its own attestation (`release.yml:350–353`), so it is not unprotected, but\n\n**the instinctive \"download everything and run `sha256sum -c`\" does not reach it.**\n\n**Also going to open an issue.**\n\n**3. `docs/release-process.md` lags the pipeline.** It tells you to *copy* the real\n\ntag, commit, digests, SBOM checksums and attestation URLs into an evidence document\n\nand then run the verifier (`:24`–`:32`). The pipeline has been doing that\n\nautomatically for a while, and uploads the result as a release asset. Impact: a reader\n\nconcludes the manifest is hand-written after the fact, which makes it look weaker than\n\nit is. **Also going to open an issue.**\n\n**4. Superseded builds are never cleaned up.** The two images from section 9 are still\n\npullable, still verifiable, still runnable. We have no \"delete same-tag intermediates\n\nafter a successful release\" step. Impact: as described. Workaround: pull the digest\n\nfrom `release-artifacts.json`.\n\n**5. The verifier checks shape, not truth.** `provenance_attestation` only has to be a\n\nnon-empty string (`:58`, `:64`, `:106`, all via `_require_text`). The script never\n\nfetches the URL and never compares the attested subject digest to the one in the\n\nmanifest. Impact: a well-formed manifest full of fabricated URLs passes. Workaround:\n\nit is a **structural** check, not a **trust** check — the trust comes from\n\n`gh attestation verify`, and you need both. Defensible division of labour, but the\n\ndocs never say so.\n\n**6. `REQUIRED_IMAGES` is hard-coded.** `verify_release_artifacts.py:15` pins\n\n`{\"server\", \"knowledge-worker\", \"web\"}` and line 114 demands exact equality. Add a\n\nfourth image and the release breaks at the final step. Impact: a bad surprise on\n\nrelease day. Workaround: know it is there — and the case it blocks (silently shipping\n\nwith an image missing) is worth more than the nuisance.\n\n**7. No process for SBOM false positives.** Sections 7 and 8. Nobody owns annotating\n\n`Simple Launcher` as noise, so the next reader has to work it out again. Impact: the\n\nsignal-to-noise ratio degrades as images grow.\n\n**8. I could not identify `e7c993b9…`.** That is where section 8 stops. **This entry exists so I do not pretend otherwise**: I can confirm it is absent from the registry\n\n`8105cae…`). Later\nreleases will have different values; the method does not change.`gh attestation verify`\n**A passing attestation says \"these bytes came out of that pipeline\"; it does not say \"this is the release.\"** What joins the two is a manifest binding tag, commit and\n\n`release-artifacts.json`, watched by a 178-line script whose hardest\nThe repository is `github.com/soit-ai/soit`, and every step above is reproducible:\n\n`curl` calls in section 2 need no account — an anonymous token is enough;`3a5b3b1a…` image from section 9 is still there; verify it yourself and watch\nit exit 0;`-c core.autocrlf=false`.\nIf any claim here is wrong — especially the digest in section 8 that I failed to\n\nidentify — open an issue and say so. I would rather learn where this is wrong than be\n\ntold the pipeline looks thorough.", "url": "https://wpnews.pro/news/gh-attestation-verify-said-yes-to-an-image-we-never-released", "canonical_source": "https://dev.to/judezh/gh-attestation-verify-said-yes-to-an-image-we-never-released-c82", "published_at": "2026-09-13 22:38:41+00:00", "updated_at": "2026-09-13 23:21:17.674519+00:00", "lang": "en", "topics": ["ai-infrastructure", "developer-tools", "mlops"], "entities": ["soit-ai", "GitHub Container Registry", "BuildKit", "Sigstore", "SLSA", "SPDX", "OCI"], "alternates": {"html": "https://wpnews.pro/news/gh-attestation-verify-said-yes-to-an-image-we-never-released", "markdown": "https://wpnews.pro/news/gh-attestation-verify-said-yes-to-an-image-we-never-released.md", "text": "https://wpnews.pro/news/gh-attestation-verify-said-yes-to-an-image-we-never-released.txt", "jsonld": "https://wpnews.pro/news/gh-attestation-verify-said-yes-to-an-image-we-never-released.jsonld"}}