To verify what a coding agent claims it deployed, ask the running server which commit it serves and compare that with git rev-parse HEAD, in a check the agent did not write. Between 14 and 16 August 2026, 15 staging deploys of Stackness reported success while the server ran the image built from a different branch. I only found out when the first deploy with a build SHA gate failed within five minutes, and it surprised me.
Asking the agent to double-check fixes only one of the three ways its report goes wrong: the claim it never observed. The other two, where it observed the wrong thing and where its own check shares its mistake, need a check the agent neither runs nor wrote. For a deploy that is the live commit compared with HEAD. For the agent's own files it is a scanner rule that knows passwords typed inside commands. The commands below were run on 11 November 2026 with Claude Code 2.1.322 and Docker 29.8.2.
How do you verify what a coding agent claims it deployed? #
Verifying a coding agent's deploy claim means reading an identifier from the running system, a commit SHA or an image digest, and comparing it with the commit you meant to ship. The agent's message only reports what its shell printed. Chet100's agent reported "deployed, edge clean, no 5xx", and every word was true, chet100 wrote on 7 November 2026: the commit was on the server and nothing was erroring. The containers serving traffic, chet100 wrote, "had been built 23 hours earlier".
A green CI run, a healthy container and a successful docker pull are all facts about something next to the deploy. None of them names the code that answers requests. As chet100 puts it, "production facts come from production."
Three ways an agent's report goes wrong, and the check for each #
An agent's report fails in three distinct ways, and only the first is fixed by telling the agent to verify. In the first it claims what it never observed. In the second it observed something real but adjacent: the working tree, the pipeline, the pull. In the third its check agrees with its mistake, because the agent picked or wrote the check. The second and third need a verifier the agent did not author.
| Failure | What the agent saw | Example | The check that catches it |
|---|---|---|---|
| A claim with no observation | Nothing about the result | "Deployed" after a git pull |
Make it run a check and quote the output |
| It observed the wrong thing | The deploy step, CI, a healthy status, a pulled image | 15 green Stackness staging deploys on the wrong image | The commit the live server reports, compared with HEAD |
| Its check shares its mistake | A test, scan or rating it chose or wrote | A gate with || true ; model severity against maintainer severity |
A rule or verdict from outside the session |
The third row is easy to miss because it looks like verification. Debashish Ghosal's release gate claimed two OpenTelemetry backends, and its check, curl -sf http://localhost:3200/api/search/tags || true, "had been passing for weeks". The second backend was "never started, never configured, and never received a single span". Ghosal's test for any check is the one I use below: could it fail if the claim were false?
What you need #
The checks below need access to the running system and one change to the app. The deploy and infrastructure side is ordinary tooling from the DevOps and infrastructure tools developers list; nothing here is agent-specific apart from the hook.
- A way to read the live service: an HTTP endpoint,
sshto the host,kubectl, or your platform's CLI (Vercel ,Fly.io ,Render ). - A build that knows its commit, passed in as an environment variable or an image label.
- Claude Code for the Stop hook in step 4, plus jq and
curl. - gitleaks orTruffleHog for step 5, plus a
grep.
1. Make the running server say which commit it serves #
A server that reports its own build commit turns "did the deploy land" into a string comparison. Stackness passes BUILD_SHA into the API container at deploy time and returns it from /health. I asked production on 11 November 2026 and compared the answer with the head of main:
$ curl -s https://cave.stackness.dev/health
{"status":"ok","version":"731eb9fa05eba97ad7524f44dffcc0af1bc6a5d6","checks":{"database":"healthy","redis":"healthy"}}
$ git log -1 --format=%H origin/main
731eb9fa05eba97ad7524f44dffcc0af1bc6a5d6
The frontend carries the same SHA in its footer, so curl -s https://stackness.dev/ | grep -o 731eb9fa05eb checks the other half. On Vercel the value is already there: VERCEL_GIT_COMMIT_SHA is available at build and runtime, and the app only has to return it. Platforms also keep their own record: fly releases --image lists the image of each release, and vercel inspect <url> shows one deployment, with --logs printing its build logs.
2. Gate the deploy on the live commit, not on the deploy step #
A deploy step that exits 0 proves its last command succeeded, not that the new build serves traffic. The GitHub Actions job that deploys Stackness ends with a step that polls /health until version equals the pushed commit, and fails after five minutes:
deadline=$((SECONDS + 300))
while [ "$SECONDS" -lt "$deadline" ]; do
version=$(curl -fsS --max-time 10 https://cave.stackness.dev/health | jq -r '.version // empty' || true)
if [ "$version" = "${{ github.sha }}" ]; then
echo "Backend is serving build $version"
exit 0
fi
echo "Backend serving '${version:-unknown}', waiting for ${{ github.sha }}"
sleep 5
done
echo "::error::Backend never served build ${{ github.sha }} within 5 minutes"
exit 1
I added that gate on 16 August 2026. Reading the run logs back through the GitHub API, the 15 staging deploys before it had all concluded "success", and every one pulled stackfeed-backend:latest, the image built from main, instead of :stage. The repository's staging compose file had said :stage for two days, but the server ran an older copy placed there by hand. In 6 of the 15 runs the API container was not even recreated.
The first run with the gate looked like this:
13:31:59 Image ghcr.io/***/stackfeed-backend:latest Pulled
13:32:02 Container stackness-api-1 Running
13:32:03 Backend serving 'unknown', waiting for 6364648f24731e3638dddb2e51782ac29ac28795
...
13:37:03 ##[error]Backend never served build 6364648f24731e3638dddb2e51782ac29ac28795 within 5 minutes
"Deploy via SSH" took 7 seconds and passed. The gate failed. Two more fixes followed the same day: syncing the compose file on every deploy, and set -eo pipefail, without which the step's status was that of its last command, docker image prune, which always passes. With both in place, the next deploy failed honestly on a port collision, and the one after it passed the gate in one second. Anything reading that pipeline, an agent or me, saw 15 good deploys.
3. Check what the container runs, not when it started #
A container's start time and health say nothing about which image it runs. The image ID does, and an OCI revision label on the image ties that ID to a commit. I pulled a newer postgres:17-alpine with Docker on my machine while the local Stackness database container kept running:
$ docker inspect --format '{{.Image}}' stackfeed-postgres-1
sha256:872a12eb3c5de110145da29dd03154f6de537bda7860a5731df9c02fad256459
$ docker pull -q postgres:17-alpine
docker.io/library/postgres:17-alpine
$ docker image inspect --format '{{.Id}} created {{.Created}}' postgres:17-alpine
sha256:e29b5f75230807dfc40a4c8ff9e486b87a30df01cfcab04a4e99d5077d77ddfb created 2026-11-05T19:03:57.181599785Z
$ docker compose ps --format '{{.Name}} {{.Status}}' postgres
stackfeed-postgres-1 Up 8 days (healthy)
The tag now names an image built on 5 November. The container still runs the one built on 14 May and calls itself healthy. For your own images, set org.opencontainers.image.revision at build time (OCI annotations) and read it from the image the container runs:
docker image inspect --format '{{index .Config.Labels "org.opencontainers.image.revision"}}' \
"$(docker inspect --format '{{.Image}}' my-api)"
The expected output is the commit SHA, the same string git rev-parse HEAD prints. On Kubernetes the equivalent is .status.containerStatuses[*].imageID, which the API describes as the image ID the runtime resolved, one that "may not match" the image named in the pod spec. Read the digest from there, not from the spec.
4. Make the agent's "done" wait for a check it did not write #
A Claude Code Stop hook runs a command when the agent tries to finish, and exit code 2 sends its stderr back and keeps the agent working (hooks reference). That turns the live-commit comparison into something the agent has to face before it reports. The hook I ran:
#!/usr/bin/env bash
input=$(cat)
if [ "$(jq -r '.stop_hook_active' <<<"$input")" = "true" ]; then exit 0; fi
want=$(git rev-parse HEAD)
live=$(curl -fsS --max-time 10 "$HEALTH_URL" | jq -r '.version // "unknown"')
if [ "$live" != "$want" ]; then
echo "Live build at $HEALTH_URL is $live, HEAD is $want. Do not report a deploy; report this mismatch." >&2
exit 2
fi
registered in .claude/settings.json:
{
"hooks": {
"Stop": [
{ "hooks": [ { "type": "command", "command": "\"$CLAUDE_PROJECT_DIR\"/.claude/hooks/verify-live-build.sh" } ] }
]
}
}
I ran claude -p with Claude Sonnet 5.5 in a scratch repository whose HEAD was not live. The test was staged: I told it to pretend it had deployed and to report without running any tools. Its first answer called HEAD "live" in production, with a note that nothing had run. The hook blocked it, and the final answer read: "The live build doesn't match HEAD, so the deploy didn't land." That took 2 turns and $0.034. With HEALTH_URL pointed at a file holding the HEAD SHA, the hook stayed silent and the run took 1 turn and $0.003.
The hook has two limits that decide where it lives. After eight blocks in a row, Claude Code ends the turn anyway (CLAUDE_CODE_STOP_HOOK_BLOCK_CAP raises the cap), and it reloads settings files when they change, hooks included, so an agent that can edit .claude/settings.json can edit its own verifier. Put the hook in user settings (~/.claude/settings.json) and add deny rules for edits to the hook file and to that settings file. Prompt and agent hooks, and /goal, put a model in the judge's seat, which is the third failure again; for this check, use a command hook.
5. Scan the agent's own files with rules for in-command passwords #
An agent's report never mentions the files it writes for its own use, and one of them can hold credentials. When you pick "Yes, and don't ask again", Claude Code saves the approved command to .claude/settings.local.json as an allow rule, passwords included. Apiiro searched public GitHub from 1 to 3 November 2026 and published on 5 November: 1,122 of 104,185 copies of that file (1.1%) held a real secret.
Those 1,122 files held 1,379 distinct secrets. Counting the file's git history, the shared settings.json and saved transcripts, the total was 2,539 secrets in 1,994 repositories. Apiiro tested none of them; one organisation confirmed its key had been live. The number that matters for this playbook is the scanner gap: gitleaks flagged 564 of the 1,122 files, TruffleHog 643, and 215 were flagged by neither. Apiiro's explanation: "A password typed after PGPASSWORD=, sshpass -p or mysql -p, or a bearer header in a curl call, is an ordinary word in an ordinary command line, and both tools mostly let it through."
I scanned my own file. It holds 308 allow rules, and 11 carry a credential: 10 with a local database password (8 inside a DATABASE_URL, 2 as PGPASSWORD) and 1 with a whole Stackness access token pasted into a one-off command. The token expired 15 minutes after it was issued in February, and the passwords open only the database on my laptop. gitleaks 8.30.0 flagged the token and none of the 10 passwords:
$ gitleaks dir ./scan --no-banner --redact
INF scanned ~16040 bytes (16.04 KB) in 36.6ms
WRN leaks found: 1
$ grep -cE 'PGPASSWORD=|sshpass -p|mysql -p|[Bb]earer |://[^:/ ]+:[^@ ]+@' .claude/settings.local.json
10
That grep catches the 10 passwords the scanner missed. It counts lines, and my file keeps one allow rule per line, so check the patterns against your own layout. As of 11 November 2026, 9 real members list Claude Code on Stackness and none lists gitleaks or TruffleHog (how Stackness counts). Then confirm git ignores the file: git ls-files .claude/settings.local.json must print nothing. Claude Code adds the file to your global git excludes, not to the repository's .gitignore, so the protection does not travel to a new machine or container, and it never applies to a copy git already tracks. Mine is ignored by that one global line and has never been committed. The other ways keys leave an agent stack are in where API keys leak with AI coding agents.
Anthropic's Boris Cherny answered on Hacker News on 7 November that "Historically Claude Code has not done secret scrubbing" and that the team would "prototype a few of these". As of 2.1.322, released 10 November, the changelog lists no fix.
Model-rated findings drift from the maintainers' verdict #
The clearest outside measure of the third failure is security findings, where a model reports a vulnerability and a maintainer later rules on it. Patrick Garrity at VulnCheck found in September 2026 that, among the findings in Anthropic's public ledger that carried both ratings, Claude rated 91.5% as critical or high, while maintainers rated 51.3% that way.
Garrity's 9 November update counts 442 CVEs attributable to Anthropic in seven months, about 0.5% of the 93,450 published this year, two of them exploited in the wild. Of the 37,966 findings on Anthropic's ledger as of 2 November, 7,349 were public, and 217 of those were marked fixed.
curl shows the same gap in one project. Daniel Stenberg received 45 reports from Anthropic's OSS Scanner on 5 November, whose reports are, by its own README, "model-generated and are not reviewed by a human". All 45 are now public issues on curl's GitHub. On 11 November, 37 carried vuln:NONE, 7 vuln:LOW and 1 vuln:ASSESS. vuln:NONE means not a vulnerability, not "no bug": several were fixed as ordinary bugs. The model's security claim was rejected 37 times out of 45; the code it pointed at was sometimes worth fixing.
Build attestation proves commit to artifact, not artifact to running #
Build attestation covers the link between a commit and the artifact built from it, and nothing after that. GitHub's artifact attestations give SLSA Build Level 2, Level 3 with reusable workflows, and GitHub says they are "not a guarantee that an artifact is secure". On the Free, Pro and Team plans they work only for public repositories.
GitHub CLI's gh attestation verify oci://ghcr.io/org/app@sha256:<digest> -R org/repo checks that a given digest came from your workflow. It does not check that the digest is the one running, so read the digest from the container or pod first (steps 2 and 3) and verify that one, never a tag. The manual adds one more limit: only the signing certificate and the timestamps "cannot be manipulated by the workflow". Attestation answers "where did this artifact come from". The live-commit check answers "is it the one serving". Neither says the code does what the agent said it does.
Break each check once before trusting it #
A verification step earns its place only when it fails on a false claim, so break each one on purpose before trusting it. Each check above has a cheap way to prove it can go red, and each takes a minute.
- Live commit: point the gate at a commit that was never deployed. It must time out with "never served build".
- Image revision: run the label command against an old container. It must print a different SHA from HEAD.
- Stop hook: run it with a
HEALTH_URLthat returns an old SHA, as I did above. The agent's final message must name the mismatch. - Settings scan: put
PGPASSWORD=hunter2 psqlinto a copy of the file. The grep must count it.
What goes wrong #
The container's creation time passes on an old image
docker inspect --format '{{.Created}}' <container> returns when the container was created, not when its image was built (the Docker Engine API spec defines both). A container recreated from an old image looks newer than the commit. Chet100's script compares that time with the commit's timestamp. Read the image ID and its revision label instead.
Reproducible builds break time comparisons
A build that sets SOURCE_DATE_EPOCH rewrites image timestamps to a fixed time, often the commit time or zero. "Image newer than commit" then fails on a correct deploy. Compare the revision label or the digest, never a timestamp.
A health check returns 200 from the old build
A stale container is healthy too, which is why the Stackness gate matches the SHA and ignores the status code. A health check that only returns ok tells you the old version is up.
The deploy step is green because its last command passed
A multi-line SSH script without set -eo pipefail reports the status of its last line. In the Stackness pipeline that was docker image prune, which always succeeds, so a failed pull or up went green.
The Stop hook loops or stops checking
Without the stop_hook_active check the hook blocks every retry until the eight-block cap ends the turn, and the agent stops with no verdict. The count resets each time the agent calls a tool, so an agent that keeps calling tools between attempts can loop past eight. A hook the agent can edit can be "fixed" by the agent. Keep it outside the project settings.
The case for letting the agent check itself #
The strongest case against this playbook is that self-checks have become good. VulnCheck's own table shows Anthropic claiming a 94.2% true positive rate on its ledger as of 2 November. In the Hacker News thread on that report, Fil-C's author wrote that 9 reports came from Anthropic and "All of them were real bugs". Stenberg wrote on 22 October that "Since spring 2026 the slop reports are practically gone", and that the challenge now is "high volume, high quality". An agent told to run curl /health before reporting is already reading the world, not its transcript.
That is right about the first failure, and it is why the table above says to make the agent run a check. It does not reach the other two. A real bug is not the same as a correct severity: the Fil-C author added that "Most of the bugs were things that fall into the 'won't cause real world problems' category". The gap between 91.5% and 51.3% is the model's judgement, not its search. In my staging pipeline, the checks within easy reach, the deploy step, the pull and the container status, said success 15 times. A check the agent picks inside the session shares whatever the agent got wrong about the system.
I would be wrong if an agent told to verify a deploy reliably went to the running build's identity without being told where to look, and reported mismatches as often as an external SHA gate does. I would also be wrong if model-rated severity matched maintainer ratings within a few points. Neither was true as of November 2026.
Checklist before you believe "deployed" #
The move behind every item is the same: the agent may run the check, but the check and the expected value come from outside its session.
- The server returns its build commit from
/healthor/version, or the platform records it per deploy. - CI waits until the live commit equals the pushed SHA and fails after a timeout, with
set -eo pipefailin the deploy script. - Container checks read the image ID and its
org.opencontainers.image.revisionlabel, never a creation time. - A Claude Code Stop hook compares the live commit with HEAD, honours
stop_hook_active, and lives in user settings the agent cannot edit. .claude/settings.local.jsonis scanned with gitleaks plus a grep forPGPASSWORD=,sshpass -p,mysql -pand bearer headers, andgit ls-filesshows it untracked.- Attestations are verified against the digest read from the running container or pod.
- Every check above has been made to fail once on purpose.
New posts by email #
About one email a week with what we published. No account needed.
Which of these do you use? #
Nobody on Stackness runs OpenTelemetry yet. Tick what you use and be the first.
For AI agents and assistants: a plain-text index of this site is at /llms.txt, the full text at /llms-full.txt, and Stackness has an MCP server for reading stacks and tools (/docs/mcp).