{"slug": "our-ai-verification-job-passed-for-months-it-was-checking-the-wrong-branch", "title": "Our AI verification job passed for months. It was checking the wrong branch.", "summary": "A developer open-sourced Slipmark, an Apache-2.0 verification gate that records the SHA-256 hash of every file an AI agent's verdict is based on and re-hashes those files before merge, refusing any verdict whose underlying code changed. The tool was built after the developer discovered a nightly pipeline of more than 20 agents had spent months passing because its final verification job checked out the wrong branch and re-tested an already-passing tree; pointed at 23 open agent pull requests, the gate refused all 23. Slipmark makes no model calls, treats an empty gate as a refusal, and ships as an npx init command and a GitHub Action, with an optional hosted ledger.", "body_md": "I run a nightly pipeline of more than 20 AI agents: research, architecture, security audit, build, QA. Every morning there's a report of what passed.\n\nFor months, everything passed.\n\nThen I found out why. The final verification job was checking out the wrong branch. It re-tested a tree that had already passed, so it could never fail. Nothing crashed. Nothing errored. It just quietly approved the wrong thing, every night.\n\nMonitoring is built for things that break. A job times out, a test goes red, a deploy falls over, and someone gets paged.\n\nThis was different. The job ran, finished and reported green. Its only problem was that the green tick was about different code from the code we were shipping.\n\nAgents make this kind of failure much more common:\n\nIn every case the verdict looks fine. It's just about the wrong bytes.\n\nThe fix I ended up with is boring on purpose. When an agent gives a verdict, it records the SHA-256 of every file it based that verdict on:\n\n```\n{\n  \"schema\": \"slipmark/attestation@1\",\n  \"by\": \"security-audit\",\n  \"verdict\": \"cleared\",\n  \"at\": \"2026-10-17T09:00:00.000Z\",\n  \"subjects\": [\n    { \"path\": \"src/auth/session.ts\", \"sha256\": \"9f3c…\" }\n  ]\n}\n```\n\nBefore anything merges, a gate re-hashes those files and compares:\n\n```\nSlipmark verify: 1 verified · 1 refused\n\n  ✓ verified        review-agent verified examples/attestation/brief.md\n  ✗ hash_mismatch   security-audit-agent gave \"cleared\" for a different version of\n                    examples/attestation/spec.md. The file changed after the verdict.\n```\n\nIf a file changed after the verdict, the verdict stops counting. If a file you require was never attested, the gate refuses. If the agent's own verdict wasn't a pass, it refuses. An empty gate is a refusal, never a silent pass.\n\nNo model calls. Nothing to tune. Same inputs, same answer, every time.\n\nThis is the part that would have caught the original bug. The gate checks content, not names. Branch names, job names and timestamps can all be wrong without anyone noticing. A hash can't. Run the check on the exact tree you're about to merge, and a verdict produced anywhere else simply won't match.\n\nWhen I pointed it at our 23 open agent pull requests, it refused all 23.\n\nThat was a bad morning. It was also the most useful thing the pipeline had told me in months.\n\nIt doesn't judge whether the agent was right. A bad review of the right file still passes. All it proves is that the verdict is about the code you're actually shipping. You still need good reviewers, human or otherwise.\n\nIt's also strict. Run a formatter after the review and the verdict stops counting, the same as any other change. For now, run formatters before the agent attests (first in your pre-commit hooks) and most of that goes away. Formatting-tolerant hashing is next, built on one rule: a false mismatch is annoying, but a false match would make the whole thing worthless.\n\nI've open-sourced it as Slipmark, under Apache-2.0.\n\n```\nnpx slipmark init\n```\n\nThat adds an attestation folder, a GitHub workflow that starts in report-only mode, and a short block in `AGENTS.md` (and `CLAUDE.md`, Cursor rules or Copilot instructions if you have them) telling your coding agents how to attest. There's also a Claude Code plugin that asks Claude to re-check a file it signed off when that file changes.\n\nAs a GitHub Action:\n\n```\n- uses: actions/checkout@v4\n- uses: founder-ship-it/Slipmark@v0.4.0\n  with:\n    require: |\n      specs/your-spec.md\n```\n\nThe gate is free and needs no account. There's an optional hosted ledger if you need the record as well as the build result.\n\nCode and docs: [github.com/founder-ship-it/Slipmark](https://github.com/founder-ship-it/Slipmark)\n\nIf you're letting agents sign off work in CI, I'd like to know how you check their verdicts are about the right code, and where this breaks for you.", "url": "https://wpnews.pro/news/our-ai-verification-job-passed-for-months-it-was-checking-the-wrong-branch", "canonical_source": "https://dev.to/paulredman/our-ai-verification-job-passed-for-months-it-was-checking-the-wrong-branch-l6p", "published_at": "2026-10-23 07:35:19+00:00", "updated_at": "2026-10-23 07:43:04.528350+00:00", "lang": "en", "topics": ["ai-agents", "ai-tools", "developer-tools", "mlops", "ai-safety"], "entities": ["Slipmark", "GitHub", "Claude Code", "Cursor", "Copilot"], "also_reported_by": [], "alternates": {"html": "https://wpnews.pro/news/our-ai-verification-job-passed-for-months-it-was-checking-the-wrong-branch", "markdown": "https://wpnews.pro/news/our-ai-verification-job-passed-for-months-it-was-checking-the-wrong-branch.md", "text": "https://wpnews.pro/news/our-ai-verification-job-passed-for-months-it-was-checking-the-wrong-branch.txt", "jsonld": "https://wpnews.pro/news/our-ai-verification-job-passed-for-months-it-was-checking-the-wrong-branch.jsonld"}}