Pulling a container image somebody else built asks four separate questions. Most
people answer three of them and skip the fourth:
| Step | Question it answers | If you skip it |
|---|---|---|
| Fetch by digest | are these the bytes I asked for | you are gambling on a mutable tag |
| Verify provenance | who built these bytes, from what, where | unknown origin |
| Read the SBOM | what is inside | a CVE lands and you cannot tell if you are affected |
| Check the release manifest | is this digest the one that version shipped | you may be verifying a different build |
The fourth is the one that has no substitute, and it is the one that is almost
always skipped. What follows is all four, run for real against our own v1.0.0.
This is where the confusion starts. Get an anonymous pull token — a public package
needs no account:
TOKEN=$(curl -s "https://ghcr.io/token?scope=repository:soit-ai/soit/server:pull&service=ghcr.io" \
| sed -n 's/.*"token":"\([^"]*\)".*/\1/p')
Ask what the v1.0.0 tag resolves to:
curl -sI -H "Authorization: Bearer $TOKEN" \
-H "Accept: application/vnd.oci.image.index.v1+json" \
https://ghcr.io/v2/soit-ai/soit/server/manifests/v1.0.0
docker-content-digest: sha256:96b80ae141000adde27cf3dedb27935b5f5d0085d69025cffb4bbb63276d5929
Content-Type: application/vnd.oci.image.index.v1+json
Content-Length: 856
856 bytes: this is an image index, not the image. Inside it, two entries:
manifests[0]: sha256:84e7f539...d32f 2589 bytes linux/amd64
manifests[1]: sha256:a6bd430d...e1be 565 bytes unknown/unknown
vnd.docker.reference.type: attestation-manifest
vnd.docker.reference.digest: sha256:84e7f539...d32f
That unknown/unknown entry is not corruption — it is BuildKit's own attestation
for the amd64 manifest, and it is a different mechanism from the Sigstore signatures we are about to read. First thing people conflate.
One level down, the amd64 manifest says:
config: sha256:021814982fcc214a02a2b322cbc3729b3ce43bd6e191aa0fa39da540a3e3cd1f 8769 bytes
layers: 12, 684216450 bytes compressed (about 652 MiB)
Three hashes so far, three different meanings:
| Name | Value for server v1.0.0 | Refers to |
|---|---|---|
| index digest | 96b80ae1…5929 |
the multi-platform index itself |
| platform manifest digest | 84e7f539…d32f |
the linux/amd64 image |
| config digest | 02181498…cd1f |
the image config JSON |
The signature covers the first one. A fourth hash shows up in section 8, and it
is none of these.
The OCI spec defines a referrers API for "things attached to this thing":
curl -s -H "Authorization: Bearer $TOKEN" \
https://ghcr.io/v2/soit-ai/soit/server/referrers/sha256:96b80ae1...5929
{"errors":[{"code":"MANIFEST_UNKNOWN","message":"manifest unknown"}]}
Same error for the platform manifest digest. Do not conclude "unsigned" from this. The spec has a fallback: replace the colon with a dash and treat the digest
{"name":"soit-ai/soit/server","tags":[
"v1.0.0",
"sha256-3a5b3b1a2d14e0646298826602124ba22e63f8c078f653e506a0a042bfd18246",
"sha256-236201a5a3ee861ffdb5fdd7ec134454619eb1ab0e777439c4a22a65002f874f",
"sha256-96b80ae141000adde27cf3dedb27935b5f5d0085d69025cffb4bbb63276d5929"]}
The last one is ours. Remember the first two — section 9 is entirely about them.
Fetching sha256-96b80ae1… returns a small index holding two Sigstore bundles:
sha256:658b3428...80b9 812 bytes
artifactType : application/vnd.dev.sigstore.bundle.v0.3+json
predicateType: https://slsa.dev/provenance/v1
created : 2026-08-05T16:14:12.946Z
sha256:59bac8c5...d9cf 814 bytes
predicateType: https://spdx.dev/Document/v2.3
created : 2026-08-05T16:14:22.004Z
Build provenance and an SBOM attestation, signed nine seconds apart. Both manifests
carry subject pointing back at sha256:96b80ae1…5929 — the single place where "this signature belongs to that image" is actually written down.
Pull the bundle blob and hash it:
curl -sL -H "Authorization: Bearer $TOKEN" \
https://ghcr.io/v2/soit-ai/soit/server/blobs/sha256:eae5d902...dd86 -o bundle.json
sha256sum bundle.json
The hash equals the digest I asked for. A content-addressed registry means this step
verifies itself; no trust required yet.
The bundle has three top-level fields — mediaType, verificationMaterial,
dsseEnvelope. The envelope's payloadType is application/vnd.in-toto+json, and
the base64 payload is the only thing the signature covers:
{
"_type": "https://in-toto.io/Statement/v1",
"subject": [{
"name": "ghcr.io/soit-ai/soit/server",
"digest": {"sha256": "96b80ae141000adde27cf3dedb27935b5f5d0085d69025cffb4bbb63276d5929"}
}],
"predicateType": "https://slsa.dev/provenance/v1",
"predicate": {
"buildDefinition": {
"buildType": "https://actions.github.io/buildtypes/workflow/v1",
"externalParameters": {
"workflow": {
"ref": "refs/tags/v1.0.0",
"repository": "https://github.com/soit-ai/soit",
"path": ".github/workflows/release.yml"
}
},
"internalParameters": {
"github": {
"event_name": "push",
"repository_id": "910429753",
"repository_owner_id": "193298865",
"runner_environment": "github-hosted"
}
},
"resolvedDependencies": [{
"uri": "git+https://github.com/soit-ai/soit@refs/tags/v1.0.0",
"digest": {"gitCommit": "8105cae074f1f27d7916acfe02f9d4eabb63169f"}
}]
},
"runDetails": {
"builder": {"id": "https://github.com/soit-ai/soit/.github/workflows/release.yml@refs/tags/v1.0.0"},
"metadata": {"invocationId": "https://github.com/soit-ai/soit/actions/runs/31023642816/attempts/1"}
}
}
}
One complete sentence: commit 8105cae… on refs/tags/v1.0.0, built by
.github/workflows/release.yml on a GitHub-hosted runner in run 31023642816,
produced the image with digest 96b80ae1….
I had the repository on disk, so I closed the loop immediately:
git rev-parse v1.0.0^{commit}
Match. Note repository_id and repository_owner_id: numeric IDs are worth more
than the repository name, because names can be renamed and transferred, IDs cannot.
verificationMaterial.certificate.rawBytes is a 1735-byte DER certificate:
issuer=O = sigstore.dev, CN = sigstore-intermediate
subject=
notBefore=Aug 5 16:14:11 2026 GMT
notAfter =Aug 5 16:24:11 2026 GMT
X509v3 Subject Alternative Name: critical
URI:https://github.com/soit-ai/soit/.github/workflows/release.yml@refs/tags/v1.0.0
Two things worth stopping on.
The subject is empty. The field that traditionally names the holder is blank; the
identity moved entirely into that SAN URI, and it is not a person or an organisation —
it is a workflow at a ref.
The validity window is exactly ten minutes. That is keyless signing: the pipeline
trades an OIDC token for a short-lived certificate, signs, and throws the key away.
Nothing to store, nothing to rotate, nothing to steal later. Ten minutes on, the
certificate is expired — and the signature still verifies, for the reason in section 5.
The certificate also carries a block of Sigstore extensions describing the build. The
values below are what I read; I am deliberately not putting names on the OIDs —
those live in Fulcio's OID documentation, and the values speak for themselves:
| OID suffix | Value |
|---|---|
.1.1 /.1.8 |
https://token.actions.githubusercontent.com |
.1.2 /.1.20 |
push |
.1.3 /.1.10 /.1.13 /.1.19 |
8105cae074f1f27d7916acfe02f9d4eabb63169f |
.1.4 |
release |
.1.5 |
soit-ai/soit |
.1.6 /.1.14 |
refs/tags/v1.0.0 |
.1.9 /.1.18 |
…/release.yml@refs/tags/v1.0.0 |
.1.11 |
github-hosted |
.1.12 |
https://github.com/soit-ai/soit |
.1.15 /.1.17 |
910429753 /193298865 |
.1.16 |
https://github.com/soit-ai |
.1.21 |
…/actions/runs/31023642816/attempts/1 |
.1.22 |
public |
.1.24 |
repo:soit-ai/soit:ref:refs/tags/v1.0.0 |
Extended Key Usage is Code Signing, Key Usage is Digital Signature only, and
there is a CT precertificate SCT timestamped Aug 5 16:14:11.958 2026 GMT — the act
of issuing this certificate was itself logged publicly.
verificationMaterial.tlogEntries holds exactly one record:
logIndex : 2346649359
integratedTime : 1785946452 -> 2026-08-05T16:14:12Z
kindVersion : {kind: dsse, version: 0.0.1}
That record lives in Rekor, the public append-only transparency log, and its job is
to witness when. The question a verifier asks is not "is this certificate valid
now" — it expired long ago — but "was it valid at the moment of signing". The log
pins that moment at 16:14:12Z, inside the 16:14:11– 16:24:11 window.
A detail from the actual bundle: timestampVerificationData is an empty object.
No RFC 3161 timestamp; the Rekor entry carries the time on its own. Not wrong, just
worth knowing what your trust actually rests on.
The second bundle's predicate is a complete SPDX document. The bundle blob is
3437645 bytes:
spdxVersion : SPDX-2.3
dataLicense : CC0-1.0
name : ghcr.io/soit-ai/soit/server
creationInfo : creators ["Organization: Anchore, Inc", "Tool: syft-1.42.3"],
created "2026-08-05T16:14:08Z"
packages : 710
relationships : 2840
Those 710 packages by purl type: 469 pkg:deb, 217 pkg:pypi, one
pkg:generic (python 3.11.15) and one pkg:oci (the image itself).
The 469 Debian packages come from the base image — server/Dockerfile starts at
python:3.11 — and only 217 are our own Python dependencies. That ratio is itself a
finding: two thirds of what you just signed, you did not write.
All three images side by side show the split clearly:
| Image | Packages | Composition | SBOM generated |
|---|---|---|---|
server |
710 | 469 deb + 217 pypi | 16:14:08Z |
knowledge-worker |
775 | 469 deb + 281 pypi | 16:22:23Z |
web |
1331 | 1311 npm + 18 apk | 16:15:01Z |
The 64 extra pypi packages in knowledge-worker include torch, transformers and
nvidia-cublas-cu12. The two images are two targets of one Dockerfile; the entire
difference is uv sync --extra knowledge-worker (server/Dockerfile:19 vs :27).
web is node:24-alpine, so the Debian packages become 18 apk ones.
An SPDX document normally carries a files section recording which files each
package was identified from. All three of ours have none — and it is deliberate,
right there in the workflow (.github/workflows/release.yml:132–149):
jq 'del(.files)
| .packages |= map(del(.hasFiles))
| .relationships |= map(select(
((.spdxElementId // "") | startswith("SPDXRef-File") | not)
and ((.relatedSpdxElement // "") | startswith("SPDXRef-File") | not)))' \
"$f" > "$f.tmp"
mv "$f.tmp" "$f"
size="$(stat -c%s "$f")"
test "$size" -le 16000000
The comment above it gives the reason: per-file SPDX entries push an ML-heavy image's
SBOM past the 16 MiB subject limit of actions/attest. In plain terms, the SBOM had to be shrunk until it could be signed.
The cost is real. You keep the package inventory and the package-to-package
relationships; you lose "which file did this package come from", which is exactly what
you want when a CVE lands and you need to locate it. The test -le 16000000 line is a
hard gate: if a future image blows past the ceiling the release fails there rather
than shipping an SBOM that cannot be signed. That part I think is right — better a
broken build than a half-signed artifact.
Two things I found in ours.
Five Windows launchers in a Linux image. Among the 710 packages,
Simple Launcher 1.1.0.14 appears five times, each with a CPE and no purl:
SPDXRef-Package-binary-Simple-Launcher-d60858f6579e7bb1
versionInfo : 1.1.0.14
externalRefs: cpe:2.3:a:Simple_Launcher:Simple_Launcher:1.1.0.14:*:*:*:*:*:*:*
Syft's binary classifier found them: Python packaging wheels ship small Windows
launcher executables, which will never execute inside this Linux image but are
inventoried all the same. And because the files section was stripped (section 7),
you cannot even find out where they are.
A fourth hash that matches nothing. The pkg:oci entry representing the image
itself reads:
pkg:oci/ghcr.io%2Fsoit-ai%2Fsoit%2Fserver@sha256%3Ae7c993b9ac5d7322058e0169c678b4ce21fe42fac72c5d3374404bf4642939ea?arch=amd64
e7c993b9… is not the index digest, not the platform manifest digest and
not the config digest. Asked for directly, the registry returns 404. I could not
determine what it is — the reasonable guess is an identifier the scanner computes
locally — so I am reporting the observation and not a conclusion. The practical rule
is clear enough: do not reconcile an SBOM's internal digest against the signature's subject. The binding lives in exactly one place,
subject.digest, and that value is 96b80ae1…, matching the registry exactly. v1.0.0 images, all of them verifiable
Back to those two extra tags. Fetched the same way:
| Image digest behind the fallback tag | Attestations attached | Signed at |
|---|---|---|
3a5b3b1a…8246 |
provenance only | 2026-08-05T15:41:15Z |
236201a5…f874f |
provenance + SBOM | 15:52:32Z / 15:52:44Z |
96b80ae1…5929 |
provenance + SBOM | 16:14:12Z / 16:14:22Z |
All three provenance statements name refs/tags/v1.0.0 and the same signing identity.
The commits differ:
| Image digest | Commit in provenance | Actions run | Commit subject |
|---|---|---|---|
3a5b3b1a… |
0dacfc52… |
31020921135 | ci(release): create the artifacts directory before image SBOM generation |
236201a5… |
ec822c63… |
31021873557 | ci(release): catalog packages only in image SBOMs |
96b80ae1… |
8105cae0… |
31023642816 | ci(release): trim image SBOMs to package level before attestation |
All three commits are on main locally, 23:33 / 23:44 / 00:05 in my timezone. The
story reads itself off the commit subjects: the tag was re-pointed twice that night while the SBOM step was being fixed — and each failed attempt had already
So this happens:
gh attestation verify \
oci://ghcr.io/soit-ai/soit/server@sha256:3a5b3b1a...8246 \
--repo soit-ai/soit --format json
exit code : 0
subject : 3a5b3b1a...8246
predicate : https://slsa.dev/provenance/v1
ref : refs/tags/v1.0.0
commit : 0dacfc5219c4ae57d024f9cb4208e1efad5f0d17
san : https://github.com/soit-ai/soit/.github/workflows/release.yml@refs/tags/v1.0.0
tlog : rekor.sigstore.dev @ 2026-08-05T23:41:11+08:00
Exit 0. Nothing is wrong with the signature. It states honestly what it is: an
image built by our release workflow, on refs/tags/v1.0.0, from commit 0dacfc52….
The one thing it does not say is that it is not the image we shipped.
No attack here — we built all three. But swap in a nastier scenario and it holds
immediately: a pipeline compromised once, fixed, re-tagged, with the poisoned
intermediate still sitting in the registry. Attestation will not catch that.
gh attestation verify answers "did this come out of that pipeline". It never
promised to answer "is this the release".
That answer is not in the signature. It is an asset on the GitHub Release, which
carries seven files:
soit-1.0.0.tar.gz 3148831
SHA256SUMS 433
release-artifacts.json 2877
server.spdx.json 3313874
knowledge-worker.spdx.json 4004516
web.spdx.json 3862614
source.spdx.json 3694033
release-artifacts.json is the manifest:
{
"featureKey": "release.artifacts",
"schemaVersion": 1,
"version": "1.0.0",
"release_tag": "v1.0.0",
"commit": "8105cae074f1f27d7916acfe02f9d4eabb63169f",
"clean_worktree_at_tag": true,
"images": [{
"component": "server",
"name": "ghcr.io/soit-ai/soit/server",
"release_tag": "v1.0.0",
"digest": "sha256:96b80ae141000adde27cf3dedb27935b5f5d0085d69025cffb4bbb63276d5929",
"reference": "ghcr.io/soit-ai/soit/server@sha256:96b80ae1...5929",
"sbom": {"format": "spdx-json", "path": "server.spdx.json", "sha256": "b83c3bc7...ef68", "attestation": "..."},
"provenance_attestation": "..."
}]
}
It says the thing the signature would not: v1.0.0 is 96b80ae1…, not the other
two. It is assembled by jq inside the publish-release job
(release.yml:300–347) and validated on the spot. I fed the copy downloaded from the
public internet straight into the same script that lives in the repository:
{
"commit": "8105cae074f1f27d7916acfe02f9d4eabb63169f",
"images": ["knowledge-worker", "server", "web"],
"passed": true,
"release_tag": "v1.0.0"
}
exit=0
178 lines, standard library only, no network
(server/scripts/verify_release_artifacts.py). It insists that:
release_tag equals v + version (:42); commit is 40 lowercase hex characters and not all zeroes (:45); clean_worktree_at_tag is true (:49); reference name@digest``:91) — a name:tag reference is
rejected outright;digest matches ^sha256:[0-9a-f]{64}$ (:88);:15, :114);:107–112).
A test guards that last-but-two rule: it rewrites the example's reference into tag
form and asserts the script fails with digest-pinned
(server/tests/unit/test_release_operations_contract.py:85–88).
The release also ships a git archive tarball (release.yml:253–263) described as
deterministic. Worth testing rather than believing. First, the checksum:
grep "soit-1.0.0.tar.gz" SHA256SUMS | sha256sum -c -
Then rebuild it locally:
git archive --format=tar.gz --prefix="soit-1.0.0/" \
--output=local.tar.gz 8105cae074f1f27d7916acfe02f9d4eabb63169f
c11179c379ba7390c215133f342c92a7517a548f1536959cb43e187b16a7bab3 *local.tar.gz
5d3dd50f491a897ba9a184ad6dc474e2ef55508224404266d91b3e1e1d32ae9d *soit-1.0.0.tar.gz
Different — and not just at the gzip layer; the inner tars differ too. Comparing
members explains it:
members : 1551 local / 1551 released -- same
only-one-side: 0 / 0 -- same
differing : 1280 members
mtime : 1785945959 everywhere -- same
mode : 0664 everywhere -- same
example: soit-1.0.0/.github/workflows/quality.yml local 15270 released 14803
Sizes only, and each delta equals that file's line count. CRLF. My machine has
core.autocrlf=true, so git archive helpfully rewrote the line endings. Turn it
off:
git -c core.autocrlf=false archive --format=tar.gz --prefix="soit-1.0.0/" \
--output=local2.tar.gz 8105cae074f1f27d7916acfe02f9d4eabb63169f
Identical. A Windows box, git 2.55.0, a month later, reproducing the tarball
GitHub Actions produced.
Both halves matter. Determinism is real — git archive writes no build timestamp and
takes mtimes from the commit — but it is sensitive to local git configuration, and
the first suspect when reproduction fails is core.autocrlf, not the publisher.
Everything above, compressed into something you can paste:
docker buildx imagetools inspect ghcr.io/soit-ai/soit/server:v1.0.0 | head -3
gh attestation verify oci://ghcr.io/soit-ai/soit/server:v1.0.0 --repo soit-ai/soit
gh attestation verify oci://ghcr.io/soit-ai/soit/server:v1.0.0 --repo soit-ai/soit \
--predicate-type https://spdx.dev/Document/v2.3
curl -sLO https://github.com/soit-ai/soit/releases/download/v1.0.0/release-artifacts.json
curl -sLO https://github.com/soit-ai/soit/releases/download/v1.0.0/SHA256SUMS
sha256sum -c SHA256SUMS
docker pull ghcr.io/soit-ai/soit/server@sha256:96b80ae1...5929
Step 3 deserves emphasis: gh attestation verify checks provenance only by
default. Without --predicate-type it returned exactly one result here, the SLSA
provenance. With it, the SPDX attestation comes back — 6022876 bytes of JSON holding
the same 710 packages counted in section 6. A lot of people assume "verified" includes
the SBOM. It does not.
Step 4 is the point of this whole piece. Without it, the green checkmark from step 2
shines just as brightly on the 3a5b3b1a… image from section 9.
House rule: the last section is about our own problems.
1. Our own compose pins tags, not digests. All six services in
docker/docker-compose.images.yml use :${SOIT_IMAGE_TAG:-v1.0.0} (:17–:32), and
the README says the same (README.md:146). We demand digest-pinned references inside
release-artifacts.json and then hand users a tag. Impact: a tag can be re-pointed.
Workaround: step 5 of the checklist. I plan to open an issue; none filed at the
time of writing, hence no link.
2. SHA256SUMS does not cover release-artifacts.json. The 433-byte file lists
five entries — four SPDX documents and the tarball. Neither the manifest nor
SHA256SUMS itself is in there (the latter cannot be, by construction). The manifest
does get its own attestation (release.yml:350–353), so it is not unprotected, but
the instinctive "download everything and run sha256sum -c" does not reach it.
Also going to open an issue.
3. docs/release-process.md lags the pipeline. It tells you to copy the real
tag, commit, digests, SBOM checksums and attestation URLs into an evidence document
and then run the verifier (:24–:32). The pipeline has been doing that
automatically for a while, and uploads the result as a release asset. Impact: a reader
concludes the manifest is hand-written after the fact, which makes it look weaker than
it is. Also going to open an issue.
4. Superseded builds are never cleaned up. The two images from section 9 are still
pullable, still verifiable, still runnable. We have no "delete same-tag intermediates
after a successful release" step. Impact: as described. Workaround: pull the digest
from release-artifacts.json.
5. The verifier checks shape, not truth. provenance_attestation only has to be a
non-empty string (:58, :64, :106, all via _require_text). The script never
fetches the URL and never compares the attested subject digest to the one in the
manifest. Impact: a well-formed manifest full of fabricated URLs passes. Workaround:
it is a structural check, not a trust check — the trust comes from
gh attestation verify, and you need both. Defensible division of labour, but the
docs never say so.
6. REQUIRED_IMAGES is hard-coded. verify_release_artifacts.py:15 pins
{"server", "knowledge-worker", "web"} and line 114 demands exact equality. Add a
fourth image and the release breaks at the final step. Impact: a bad surprise on
release day. Workaround: know it is there — and the case it blocks (silently shipping
with an image missing) is worth more than the nuisance.
7. No process for SBOM false positives. Sections 7 and 8. Nobody owns annotating
Simple Launcher as noise, so the next reader has to work it out again. Impact: the
signal-to-noise ratio degrades as images grow.
8. I could not identify e7c993b9…. That is where section 8 stops. This entry exists so I do not pretend otherwise: I can confirm it is absent from the registry
8105cae…). Later
releases will have different values; the method does not change.gh attestation verify
A passing attestation says "these bytes came out of that pipeline"; it does not say "this is the release." What joins the two is a manifest binding tag, commit and
release-artifacts.json, watched by a 178-line script whose hardest
The repository is github.com/soit-ai/soit, and every step above is reproducible:
curl calls in section 2 need no account — an anonymous token is enough;3a5b3b1a… image from section 9 is still there; verify it yourself and watch
it exit 0;-c core.autocrlf=false.
If any claim here is wrong — especially the digest in section 8 that I failed to
identify — open an issue and say so. I would rather learn where this is wrong than be
told the pipeline looks thorough.