{"slug": "what-green-ci-doesnt-prove-after-a-prs-base-moves", "title": "What green CI doesn’t prove after a PR’s base moves", "summary": "A developer released merge-proof, an open-source Git/GitHub Action tool that detects when a pull request's green CI does not prove the final merged state was validated against the base it merged into. The tool, which runs on local Git state with zero npm dependencies, was tested against 2,037 agent-authored PRs across 32 public repositories, finding 24.1% returned NOT_PROVEN, indicating a lack of evidence that the candidate was validated against the actual merge base.", "body_md": "A pull request can be conflict-free, have green CI, and still lack evidence that the final merged state was ever checked.\n\nThat is the narrow problem behind [merge-proof](https://github.com/ohcaygo/merge-proof), a small open-source Git / GitHub Action tool I just published.\n\nThis is not a claim that AI-authored code is broken. It is not a claim that GitHub CI is broken. It is not a claim that a flagged PR contains a bug.\n\nIt is only about evidence.\n\nIf your repository uses strict required status checks correctly, and especially if it uses a properly configured GitHub merge queue, that is the stronger answer to base drift.\n\nA merge queue tests the combined state before merge. For the specific problem of \"did this PR get validated against the base it is actually merging into?\", that is better than a diagnostic tool after the fact.\n\nmerge-proof is narrower. It is useful where that stronger enforcement is absent, incomplete, advisory, or not easy to inspect. It surfaces the evidence gap: which base commit, which candidate commit, which overlapping files, and which protected-path categories made the result unproven.\n\nIf your conclusion is \"GitHub already solves the valuable part of this for my repo,\" that is useful feedback, and I would rather hear it than not.\n\nA common PR path looks like this:\n\n`main`\n\nat commit A.`main`\n\nmoves to commit B while the PR is open.The clean merge only tells you Git could combine the text. It does not prove that CI, review, or any other validation checked that exact combination.\n\nThis matters most when the base changed in files the PR also changed. There may be no conflict, but the combined behavior was not the state represented by the earlier green check.\n\nmerge-proof currently implements two blocking checks.\n\n`BASE_DRIFT_UNVERIFIED`\n\n— the base advanced after the candidate diverged, and files changed by the candidate were also changed on the base during that interval.\n\n`PROTECTED_BOUNDARY`\n\n— the candidate changes paths classified as expensive to undo if merged without explicit evidence: migrations, schemas, auth, secrets, billing, or access-policy / RLS / RBAC areas.\n\nIt also reports `STALE_BASE`\n\nas advisory only, and reports CI and deployment configuration changes as an advisory `ci-deploy`\n\ncategory. Neither advisory changes the verdict.\n\nThe tool runs on local Git state. It has zero npm dependencies. It makes no network calls, no model calls, no API calls, and has no telemetry. In shallow clones it fails closed instead of reporting confidence from history it could not see.\n\nOne disclosure: the GitHub Action wrapper's only third-party dependency is `actions/github-script@v7`\n\n, a GitHub-owned action, used solely to post the optional sticky PR comment. The verifier itself has none.\n\nThis release does not read CI results, so it cannot establish `CI_RAN_ON_FINAL_HEAD`\n\n.\n\nIt does not read reviews or approvals.\n\nIt does not verify that a candidate is durable on a remote.\n\nIt does not compare a PR against a declared scope.\n\nIt is not a bug detector, security scanner, code reviewer, human-approval verifier, or correctness proof. A `VERIFIED`\n\nresult only means the implemented checks did not find their specific evidence gaps.\n\nWe ran merge-proof against 2,037 agent-authored PRs across 32 public repositories. 24.1% returned `NOT_PROVEN`\n\nunder the implemented checks.\n\nImmediately important: `NOT_PROVEN`\n\ndoes not mean broken or buggy. It means the repository does not contain evidence that the candidate was validated against the state it merged into. No conclusion about code correctness follows from that verdict. This is a prompt to look, not an accusation.\n\nThe exact result:\n\n`NOT_PROVEN`\n\n, or 24.1%`VERIFIED`\n\n, or 75.9%`FAIL`\n\nThe study used deterministic Git data. Agent-authored PRs were identified from commit trailers, not model judgment. The scan walked squash merges on the default branch's first-parent history, used the squash commit's first parent as the base-at-merge, fetched the PR head directly, and ran the same analyzer that ships in the package — not a separate research implementation.\n\nIt is reproducible from the repository:\n\n```\nbash\nnode study/collect.js --repos study/repos.txt --out study-data.json --max-prs 80 --prune true\nnode study/summarize.js study-data.json\n```\n\n", "url": "https://wpnews.pro/news/what-green-ci-doesnt-prove-after-a-prs-base-moves", "canonical_source": "https://dev.to/ohcaygo/what-green-ci-doesnt-prove-after-a-prs-base-moves-2c29", "published_at": "2026-09-01 21:58:53+00:00", "updated_at": "2026-09-01 22:53:21.813688+00:00", "lang": "en", "topics": ["developer-tools", "mlops"], "entities": ["merge-proof", "GitHub", "ohcaygo"], "alternates": {"html": "https://wpnews.pro/news/what-green-ci-doesnt-prove-after-a-prs-base-moves", "markdown": "https://wpnews.pro/news/what-green-ci-doesnt-prove-after-a-prs-base-moves.md", "text": "https://wpnews.pro/news/what-green-ci-doesnt-prove-after-a-prs-base-moves.txt", "jsonld": "https://wpnews.pro/news/what-green-ci-doesnt-prove-after-a-prs-base-moves.jsonld"}}