{"slug": "ci-gates-for-ai-generated-prs-need-re-derivable-evidence", "title": "CI gates for AI-generated PRs need re-derivable evidence", "summary": "Agent Gate v0.2.1 adds evidence snapshots to AI-generated pull request findings, enabling third parties to re-derive why a CI gate fired. The GitHub Action checks deterministic merge evidence without using LLMs at runtime, and the new feature provides canonical material alongside stable finding IDs for auditability. The developer behind Agent Gate aims to make failure modes visible, reproducible, and tunable before promoting warnings to merge gates.", "body_md": "When a CI gate flags an AI-generated PR, the important question is not only \"what did it flag?\"\n\nIt is also:\n\n\"Could someone else come back later and re-derive why this finding fired?\"\n\nThat is the reason I added evidence snapshots to Agent Gate v0.2.1.\n\nAgent Gate is a GitHub Action for AI-generated pull requests.\n\nIt does not review code with an LLM. It checks deterministic merge evidence in CI:\n\nThe Action does not checkout PR code, call LLMs at runtime, or execute repository scripts.\n\nIn v0.2.0, Agent Gate added stable finding IDs.\n\nThat gave every finding a short audit handle, for example:\n\n```\nagf_987ab9ddb8c1b299\n```\n\nThat is useful for references, comments, future override workflows, and log-based debugging.\n\nBut an ID by itself is not proof. If someone sees the ID later, they still need to know what recorded material produced it.\n\nv0.2.1 adds `evidenceSnapshot`\n\nto public findings.\n\nThe split is:\n\n```\nfindingId = short audit handle\nevidenceSnapshot = canonical material used to derive that handle\n```\n\nThe snapshot is intentionally boring. It contains stable rule material such as:\n\nIt does not include timestamps, report order, risk score, version, commit SHA, or mutable display text.\n\nExample compact log output:\n\n```\nAgent Gate: NEEDS HUMAN DECISION\nDecision: warn\nRisk score: 49 / 100\nWhy: Agent-generated PRs must include an agent-gate contract.\nRecommended next step: Add a PR contract before relying on scope checks.\nPolicy status: warning today; eligible to become a merge gate after tuning.\n\nFindings:\n- error agf_be0c2c2a66312aff contract/missing\n- error agf_987ab9ddb8c1b299 risk/high-risk-path .github/workflows/agent-gate.yml\n- warn agf_6016e753491255d7 workflow/dangerous-pattern .github/workflows/agent-gate.yml\n```\n\nThe compact log stays short, but the JSON and Markdown reports carry the fuller evidence.\n\nExample JSON shape:\n\n```\n{\n  \"findingId\": \"agf_987ab9ddb8c1b299\",\n  \"ruleId\": \"risk/high-risk-path\",\n  \"severity\": \"error\",\n  \"path\": \".github/workflows/agent-gate.yml\",\n  \"evidenceSnapshot\": {\n    \"ruleId\": \"risk/high-risk-path\",\n    \"severity\": \"error\",\n    \"path\": \".github/workflows/agent-gate.yml\",\n    \"evidence\": [\n      {\n        \"label\": \"changed_file\",\n        \"value\": \".github/workflows/agent-gate.yml\"\n      }\n    ]\n  }\n}\n```\n\nFor me, the bar for promoting a finding from warning to blocking is:\n\n```\nA third party should be able to re-derive the finding from recorded evidence.\n```\n\nThat does not mean the check is magically correct.\n\nIt means the failure mode is visible, reproducible, and tunable.\n\nA repo can start in warn mode, observe which findings are useful, and only later promote low-noise findings into merge gates.\n\nAgent Gate still does not prove semantic correctness.\n\nMatching test-file evidence is not proof that the tests cover the behavior. It is change evidence / self-consistency evidence.\n\nMaintainer override storage is also not implemented yet. That is probably the next hard design question: if someone bypasses a finding, where should that override live so it is durable enough to inspect later?\n\nCODEOWNERS / reviewer evidence and package dependency drift are also future work.\n\nIf you maintain a repo where coding agents open PRs, I would love feedback on whether this kind of evidence is useful or too noisy in observe mode.\n\nRepo:\n\n[https://github.com/sjh9714/Agent-Gate](https://github.com/sjh9714/Agent-Gate)\n\nDisclosure: I maintain Agent Gate. v0.2.1 is still a prerelease; I would start in warn mode before treating any finding as a merge gate.", "url": "https://wpnews.pro/news/ci-gates-for-ai-generated-prs-need-re-derivable-evidence", "canonical_source": "https://dev.to/sjh9714/ci-gates-for-ai-generated-prs-need-re-derivable-evidence-2g4e", "published_at": "2026-06-21 01:08:59+00:00", "updated_at": "2026-06-21 01:36:52.430233+00:00", "lang": "en", "topics": ["developer-tools", "ai-agents", "ai-safety"], "entities": ["Agent Gate", "GitHub", "sjh9714"], "alternates": {"html": "https://wpnews.pro/news/ci-gates-for-ai-generated-prs-need-re-derivable-evidence", "markdown": "https://wpnews.pro/news/ci-gates-for-ai-generated-prs-need-re-derivable-evidence.md", "text": "https://wpnews.pro/news/ci-gates-for-ai-generated-prs-need-re-derivable-evidence.txt", "jsonld": "https://wpnews.pro/news/ci-gates-for-ai-generated-prs-need-re-derivable-evidence.jsonld"}}