Verify what a coding agent claims it deployed: the live commit, a Stop hook and the files it wrote

A build SHA gate caught 15 staging deploys of Stackness that reported success between 14 and 16 August 2026 while the server ran an image built from a different branch, according to a first-person account that recommends comparing the commit a live server reports against `git rev-parse HEAD` in a check the coding agent did not write. The account, run on 11 November 2026 with Claude Code 2.1.322 and Docker 29.8.2, separates three failure modes in agent deploy reports — a claim never observed, an observation of the wrong thing, and a check that shares the agent's mistake — and cites chet100's 7 November 2026 report that containers serving traffic "had been built 23 hours earlier" and Debashish Ghosal's release gate whose `curl -sf http://localhost:3200/api/search/tags || true` check "had been passing for weeks" while a second OpenTelemetry backend never started.

Verify what a coding agent claims it deployed: the live commit, a Stop hook and the files it wrote To verify what a coding agent claims it deployed, ask the running server which commit it serves and compare that with git rev-parse HEAD , in a check the agent did not write. Between 14 and 16 August 2026, 15 staging deploys of Stackness reported success while the server ran the image built from a different branch. I only found out when the first deploy with a build SHA gate failed within five minutes, and it surprised me. Asking the agent to double-check fixes only one of the three ways its report goes wrong: the claim it never observed. The other two, where it observed the wrong thing and where its own check shares its mistake, need a check the agent neither runs nor wrote. For a deploy that is the live commit compared with HEAD. For the agent's own files it is a scanner rule that knows passwords typed inside commands. The commands below were run on 11 November 2026 with Claude Code https://stackness.dev/tools/claude-code 2.1.322 and Docker 29.8.2. How do you verify what a coding agent claims it deployed? Verifying a coding agent's deploy claim means reading an identifier from the running system, a commit SHA or an image digest, and comparing it with the commit you meant to ship. The agent's message only reports what its shell printed. Chet100's agent reported "deployed, edge clean, no 5xx", and every word was true, chet100 wrote on 7 November 2026 https://dev.to/chet103cc/an-agent-told-me-it-deployed-i-asked-the-server-and-the-containers-were-a-day-old-2b6i : the commit was on the server and nothing was erroring. The containers serving traffic, chet100 wrote, "had been built 23 hours earlier". A green CI run, a healthy container and a successful docker pull are all facts about something next to the deploy. None of them names the code that answers requests. As chet100 puts it, "production facts come from production." Three ways an agent's report goes wrong, and the check for each An agent's report fails in three distinct ways, and only the first is fixed by telling the agent to verify. In the first it claims what it never observed. In the second it observed something real but adjacent: the working tree, the pipeline, the pull. In the third its check agrees with its mistake, because the agent picked or wrote the check. The second and third need a verifier the agent did not author. | Failure | What the agent saw | Example | The check that catches it | |---|---|---|---| | A claim with no observation | Nothing about the result | "Deployed" after a git pull | Make it run a check and quote the output | | It observed the wrong thing | The deploy step, CI, a healthy status, a pulled image | 15 green Stackness staging deploys on the wrong image | The commit the live server reports, compared with HEAD | | Its check shares its mistake | A test, scan or rating it chose or wrote | A gate with \|\| true ; model severity against maintainer severity | A rule or verdict from outside the session | The third row is easy to miss because it looks like verification. Debashish Ghosal's release gate claimed https://dev.to/debashish ghosal/my-second-backend-existed-only-in-the-readme-e7o two OpenTelemetry https://stackness.dev/tools/opentelemetry backends, and its check, curl -sf http://localhost:3200/api/search/tags || true , "had been passing for weeks". The second backend was "never started, never configured, and never received a single span". Ghosal's test for any check is the one I use below: could it fail if the claim were false? What you need The checks below need access to the running system and one change to the app. The deploy and infrastructure side is ordinary tooling from the DevOps and infrastructure tools developers list https://stackness.dev/categories/devops-infrastructure ; nothing here is agent-specific apart from the hook. - A way to read the live service: an HTTP endpoint, ssh to the host, kubectl , or your platform's CLI Vercel https://stackness.dev/tools/vercel , Fly.io https://stackness.dev/tools/fly-io , Render https://stackness.dev/tools/render . - A build that knows its commit, passed in as an environment variable or an image label. - Claude Code for the Stop hook in step 4, plus jq https://stackness.dev/tools/jq and curl . - gitleaks https://stackness.dev/tools/gitleaks or TruffleHog https://stackness.dev/tools/trufflehog for step 5, plus a grep . 1. Make the running server say which commit it serves A server that reports its own build commit turns "did the deploy land" into a string comparison. Stackness passes BUILD SHA into the API container at deploy time and returns it from /health . I asked production on 11 November 2026 and compared the answer with the head of main : bash $ curl -s https://cave.stackness.dev/health {"status":"ok","version":"731eb9fa05eba97ad7524f44dffcc0af1bc6a5d6","checks":{"database":"healthy","redis":"healthy"}} $ git log -1 --format=%H origin/main 731eb9fa05eba97ad7524f44dffcc0af1bc6a5d6 The frontend carries the same SHA in its footer, so curl -s https://stackness.dev/ | grep -o 731eb9fa05eb checks the other half. On Vercel the value is already there: VERCEL GIT COMMIT SHA is available at build and runtime https://vercel.com/docs/environment-variables/system-environment-variables , and the app only has to return it. Platforms also keep their own record: fly releases --image lists the image of each release https://docs.fly.io/flyctl/releases , and vercel inspect <url shows one deployment https://vercel.com/docs/cli/inspect , with --logs printing its build logs. 2. Gate the deploy on the live commit, not on the deploy step A deploy step that exits 0 proves its last command succeeded, not that the new build serves traffic. The GitHub Actions https://stackness.dev/tools/github-actions job that deploys Stackness ends with a step that polls /health until version equals the pushed commit, and fails after five minutes: deadline=$ SECONDS + 300 while "$SECONDS" -lt "$deadline" ; do version=$ curl -fsS --max-time 10 https://cave.stackness.dev/health | jq -r '.version // empty' || true if "$version" = "${{ github.sha }}" ; then echo "Backend is serving build $version" exit 0 fi echo "Backend serving '${version:-unknown}', waiting for ${{ github.sha }}" sleep 5 done echo "::error::Backend never served build ${{ github.sha }} within 5 minutes" exit 1 I added that gate on 16 August 2026. Reading the run logs back through the GitHub API, the 15 staging deploys before it had all concluded "success", and every one pulled stackfeed-backend:latest , the image built from main , instead of :stage . The repository's staging compose file had said :stage for two days, but the server ran an older copy placed there by hand. In 6 of the 15 runs the API container was not even recreated. The first run with the gate looked like this: 13:31:59 Image ghcr.io/ /stackfeed-backend:latest Pulled 13:32:02 Container stackness-api-1 Running 13:32:03 Backend serving 'unknown', waiting for 6364648f24731e3638dddb2e51782ac29ac28795 ... 13:37:03 error Backend never served build 6364648f24731e3638dddb2e51782ac29ac28795 within 5 minutes "Deploy via SSH" took 7 seconds and passed. The gate failed. Two more fixes followed the same day: syncing the compose file on every deploy, and set -eo pipefail , without which the step's status was that of its last command, docker image prune , which always passes. With both in place, the next deploy failed honestly on a port collision, and the one after it passed the gate in one second. Anything reading that pipeline, an agent or me, saw 15 good deploys. 3. Check what the container runs, not when it started A container's start time and health say nothing about which image it runs. The image ID does, and an OCI revision label on the image ties that ID to a commit. I pulled a newer postgres:17-alpine with Docker https://stackness.dev/tools/docker on my machine while the local Stackness database container kept running: bash $ docker inspect --format '{{.Image}}' stackfeed-postgres-1 sha256:872a12eb3c5de110145da29dd03154f6de537bda7860a5731df9c02fad256459 $ docker pull -q postgres:17-alpine docker.io/library/postgres:17-alpine $ docker image inspect --format '{{.Id}} created {{.Created}}' postgres:17-alpine sha256:e29b5f75230807dfc40a4c8ff9e486b87a30df01cfcab04a4e99d5077d77ddfb created 2026-11-05T19:03:57.181599785Z $ docker compose ps --format '{{.Name}} {{.Status}}' postgres stackfeed-postgres-1 Up 8 days healthy The tag now names an image built on 5 November. The container still runs the one built on 14 May and calls itself healthy. For your own images, set org.opencontainers.image.revision at build time OCI annotations https://github.com/opencontainers/image-spec/blob/main/annotations.md and read it from the image the container runs: docker image inspect --format '{{index .Config.Labels "org.opencontainers.image.revision"}}' \ "$ docker inspect --format '{{.Image}}' my-api " The expected output is the commit SHA, the same string git rev-parse HEAD prints. On Kubernetes https://stackness.dev/tools/kubernetes the equivalent is .status.containerStatuses .imageID , which the API describes as the image ID the runtime resolved, one that "may not match" the image named in the pod spec. Read the digest from there, not from the spec. 4. Make the agent's "done" wait for a check it did not write A Claude Code Stop hook runs a command when the agent tries to finish, and exit code 2 sends its stderr back and keeps the agent working hooks reference https://code.claude.com/docs/en/hooks . That turns the live-commit comparison into something the agent has to face before it reports. The hook I ran: js /usr/bin/env bash Stop hook: refuse to let the agent finish while the live build is not HEAD. input=$ cat if "$ jq -r '.stop hook active' <<<"$input" " = "true" ; then exit 0; fi want=$ git rev-parse HEAD live=$ curl -fsS --max-time 10 "$HEALTH URL" | jq -r '.version // "unknown"' if "$live" = "$want" ; then echo "Live build at $HEALTH URL is $live, HEAD is $want. Do not report a deploy; report this mismatch." &2 exit 2 fi registered in .claude/settings.json : { "hooks": { "Stop": { "hooks": { "type": "command", "command": "\"$CLAUDE PROJECT DIR\"/.claude/hooks/verify-live-build.sh" } } } } I ran claude -p with Claude Sonnet https://stackness.dev/tools/claude-sonnet 5.5 in a scratch repository whose HEAD was not live. The test was staged: I told it to pretend it had deployed and to report without running any tools. Its first answer called HEAD "live" in production, with a note that nothing had run. The hook blocked it, and the final answer read: "The live build doesn't match HEAD, so the deploy didn't land." That took 2 turns and $0.034. With HEALTH URL pointed at a file holding the HEAD SHA, the hook stayed silent and the run took 1 turn and $0.003. The hook has two limits that decide where it lives. After eight blocks in a row, Claude Code ends the turn anyway CLAUDE CODE STOP HOOK BLOCK CAP raises the cap , and it reloads settings files when they change, hooks included, so an agent that can edit .claude/settings.json can edit its own verifier. Put the hook in user settings ~/.claude/settings.json and add deny rules for edits to the hook file and to that settings file. Prompt and agent hooks, and /goal , put a model in the judge's seat, which is the third failure again; for this check, use a command hook. 5. Scan the agent's own files with rules for in-command passwords An agent's report never mentions the files it writes for its own use, and one of them can hold credentials. When you pick "Yes, and don't ask again", Claude Code saves the approved command to .claude/settings.local.json as an allow rule, passwords included. Apiiro https://stackness.dev/tools/apiiro searched public GitHub https://apiiro.com/blog/coding-agents-leaking-secrets-github-dont-ask-again from 1 to 3 November 2026 and published on 5 November: 1,122 of 104,185 copies of that file 1.1% held a real secret. Those 1,122 files held 1,379 distinct secrets. Counting the file's git history, the shared settings.json and saved transcripts, the total was 2,539 secrets in 1,994 repositories. Apiiro tested none of them; one organisation confirmed its key had been live. The number that matters for this playbook is the scanner gap: gitleaks flagged 564 of the 1,122 files, TruffleHog 643, and 215 were flagged by neither. Apiiro's explanation: "A password typed after PGPASSWORD= , sshpass -p or mysql -p , or a bearer header in a curl call, is an ordinary word in an ordinary command line, and both tools mostly let it through." I scanned my own file. It holds 308 allow rules, and 11 carry a credential: 10 with a local database password 8 inside a DATABASE URL , 2 as PGPASSWORD and 1 with a whole Stackness access token pasted into a one-off command. The token expired 15 minutes after it was issued in February, and the passwords open only the database on my laptop. gitleaks https://stackness.dev/tools/gitleaks 8.30.0 flagged the token and none of the 10 passwords: bash $ gitleaks dir ./scan --no-banner --redact INF scanned ~16040 bytes 16.04 KB in 36.6ms WRN leaks found: 1 $ grep -cE 'PGPASSWORD=|sshpass -p|mysql -p| Bb earer |:// ^:/ +: ^@ +@' .claude/settings.local.json 10 That grep catches the 10 passwords the scanner missed. It counts lines, and my file keeps one allow rule per line, so check the patterns against your own layout. As of 11 November 2026, 9 real members list Claude Code on Stackness and none lists gitleaks or TruffleHog how Stackness counts https://stackness.dev/about/data-sources . Then confirm git ignores the file: git ls-files .claude/settings.local.json must print nothing. Claude Code adds the file to your global git excludes https://code.claude.com/docs/en/settings , not to the repository's .gitignore , so the protection does not travel to a new machine or container, and it never applies to a copy git already tracks. Mine is ignored by that one global line and has never been committed. The other ways keys leave an agent stack are in where API keys leak with AI coding agents https://stackness.dev/blog/where-do-api-keys-leak-with-ai-coding-agents-files-tests-and-caps . Anthropic's Boris Cherny answered on Hacker News https://news.ycombinator.com/item?id=50407882 on 7 November that "Historically Claude Code has not done secret scrubbing" and that the team would "prototype a few of these". As of 2.1.322, released 10 November, the changelog https://github.com/anthropics/claude-code/blob/main/CHANGELOG.md lists no fix. Model-rated findings drift from the maintainers' verdict The clearest outside measure of the third failure is security findings, where a model reports a vulnerability and a maintainer later rules on it. Patrick Garrity at VulnCheck https://stackness.dev/tools/vulncheck found in September 2026 https://www.vulncheck.com/blog/anthropic-glasswing-receipts that, among the findings in Anthropic's public ledger that carried both ratings, Claude rated 91.5% as critical or high, while maintainers rated 51.3% that way. Garrity's 9 November update https://www.vulncheck.com/blog/anthropic-glasswing-november-update counts 442 CVEs attributable to Anthropic in seven months, about 0.5% of the 93,450 published this year, two of them exploited in the wild. Of the 37,966 findings on Anthropic's ledger as of 2 November, 7,349 were public, and 217 of those were marked fixed. curl https://stackness.dev/tools/curl shows the same gap in one project. Daniel Stenberg received 45 reports https://www.linkedin.com/feed/update/urn:li:share:7524224297595269120/ from Anthropic's OSS Scanner on 5 November, whose reports are, by its own README https://github.com/anthropics/oss-scanner , "model-generated and are not reviewed by a human". All 45 are now public issues on curl's GitHub https://github.com/curl/curl/issues?q=is%3Aissue%20%22OSS%20Scanner%22 . On 11 November, 37 carried vuln:NONE , 7 vuln:LOW and 1 vuln:ASSESS . vuln:NONE means not a vulnerability https://daniel.haxx.se/blog/2026/11/03/scrutinize-our-assessments/ , not "no bug": several were fixed as ordinary bugs. The model's security claim was rejected 37 times out of 45; the code it pointed at was sometimes worth fixing. Build attestation proves commit to artifact, not artifact to running Build attestation covers the link between a commit and the artifact built from it, and nothing after that. GitHub's artifact attestations https://docs.github.com/en/actions/concepts/security/artifact-attestations give SLSA Build Level 2, Level 3 with reusable workflows, and GitHub says they are " not a guarantee that an artifact is secure". On the Free, Pro and Team plans they work only for public repositories https://docs.github.com/en/actions/how-tos/secure-your-work/use-artifact-attestations/use-artifact-attestations . GitHub CLI https://stackness.dev/tools/github-cli 's gh attestation verify oci://ghcr.io/org/app@sha256:<digest -R org/repo checks that a given digest came from your workflow. It does not check that the digest is the one running, so read the digest from the container or pod first steps 2 and 3 and verify that one, never a tag. The manual https://cli.github.com/manual/gh attestation verify adds one more limit: only the signing certificate and the timestamps "cannot be manipulated by the workflow". Attestation answers "where did this artifact come from". The live-commit check answers "is it the one serving". Neither says the code does what the agent said it does. Break each check once before trusting it A verification step earns its place only when it fails on a false claim, so break each one on purpose before trusting it. Each check above has a cheap way to prove it can go red, and each takes a minute. - Live commit: point the gate at a commit that was never deployed. It must time out with "never served build". - Image revision: run the label command against an old container. It must print a different SHA from HEAD. - Stop hook: run it with a HEALTH URL that returns an old SHA, as I did above. The agent's final message must name the mismatch. - Settings scan: put PGPASSWORD=hunter2 psql into a copy of the file. The grep must count it. What goes wrong The container's creation time passes on an old image docker inspect --format '{{.Created}}' <container returns when the container was created, not when its image was built the Docker Engine API spec https://github.com/moby/moby/blob/master/api/swagger.yaml defines both . A container recreated from an old image looks newer than the commit. Chet100's script compares that time with the commit's timestamp. Read the image ID and its revision label instead. Reproducible builds break time comparisons A build that sets SOURCE DATE EPOCH rewrites image timestamps https://docs.docker.com/build/ci/github-actions/reproducible-builds/ to a fixed time, often the commit time or zero. "Image newer than commit" then fails on a correct deploy. Compare the revision label or the digest, never a timestamp. A health check returns 200 from the old build A stale container is healthy too, which is why the Stackness gate matches the SHA and ignores the status code. A health check that only returns ok tells you the old version is up. The deploy step is green because its last command passed A multi-line SSH script without set -eo pipefail reports the status of its last line. In the Stackness pipeline that was docker image prune , which always succeeds, so a failed pull or up went green. The Stop hook loops or stops checking Without the stop hook active check the hook blocks every retry until the eight-block cap ends the turn, and the agent stops with no verdict. The count resets each time the agent calls a tool, so an agent that keeps calling tools between attempts can loop past eight. A hook the agent can edit can be "fixed" by the agent. Keep it outside the project settings. The case for letting the agent check itself The strongest case against this playbook is that self-checks have become good. VulnCheck's own table shows Anthropic claiming a 94.2% true positive rate on its ledger as of 2 November. In the Hacker News thread https://news.ycombinator.com/item?id=50447074 on that report, Fil-C's author wrote that 9 reports came from Anthropic and "All of them were real bugs". Stenberg wrote on 22 October https://daniel.haxx.se/blog/2026/10/22/ignore-the-slop/ that "Since spring 2026 the slop reports are practically gone", and that the challenge now is "high volume, high quality". An agent told to run curl /health before reporting is already reading the world, not its transcript. That is right about the first failure, and it is why the table above says to make the agent run a check. It does not reach the other two. A real bug is not the same as a correct severity: the Fil-C author added that "Most of the bugs were things that fall into the 'won't cause real world problems' category". The gap between 91.5% and 51.3% is the model's judgement, not its search. In my staging pipeline, the checks within easy reach, the deploy step, the pull and the container status, said success 15 times. A check the agent picks inside the session shares whatever the agent got wrong about the system. I would be wrong if an agent told to verify a deploy reliably went to the running build's identity without being told where to look, and reported mismatches as often as an external SHA gate does. I would also be wrong if model-rated severity matched maintainer ratings within a few points. Neither was true as of November 2026. Checklist before you believe "deployed" The move behind every item is the same: the agent may run the check, but the check and the expected value come from outside its session. - The server returns its build commit from /health or /version , or the platform records it per deploy. - CI waits until the live commit equals the pushed SHA and fails after a timeout, with set -eo pipefail in the deploy script. - Container checks read the image ID and its org.opencontainers.image.revision label, never a creation time. - A Claude Code Stop hook compares the live commit with HEAD, honours stop hook active , and lives in user settings the agent cannot edit. - .claude/settings.local.json is scanned with gitleaks plus a grep for PGPASSWORD= , sshpass -p , mysql -p and bearer headers, and git ls-files shows it untracked. - Attestations are verified against the digest read from the running container or pod. - Every check above has been made to fail once on purpose. New posts by email About one email a week with what we published. No account needed. Which of these do you use? Nobody on Stackness runs OpenTelemetry yet. Tick what you use and be the first. Play Lonny Eagle Amateur Skater 4A pixel art skateboarding game. Signing up opens it straight away.Sign up to play https://stackness.dev/games/lonny-eagle For AI agents and assistants: a plain-text index of this site is at /llms.txt https://stackness.dev/llms.txt?via=agent-note , the full text at /llms-full.txt https://stackness.dev/llms-full.txt?via=agent-note , and Stackness has an MCP server for reading stacks and tools /docs/mcp https://stackness.dev/docs/mcp?via=agent-note .