{"slug": "i-built-a-deterministic-ci-firewall-for-ai-generated-pull-requests", "title": "I built a deterministic CI firewall for AI-generated pull requests", "summary": "A developer built Agent Gate, a deterministic CI firewall for AI-generated pull requests that verifies merge evidence without relying on LLM judgment. The tool checks policy boundaries such as allowed file paths, workflow permission changes, and agent control plane modifications, producing a human-readable report with a clear decision. Currently in prerelease v0.1.2, Agent Gate aims to make AI-generated PRs easier to inspect before merge.", "body_md": "AI coding agents are getting good enough to open pull requests.\n\nThat is useful.\n\nIt also changes the review problem.\n\nA normal code review asks:\n\nDoes this code look correct?\n\nAn AI-generated PR also raises a different question:\n\nDid this agent change something I did not intend, and does this PR have enough evidence to merge?\n\nAgent Gate is still a prerelease, so I am starting with a narrow goal: make AI-generated PRs easier to inspect before merge.\n\nThat second question is why I built **Agent Gate for AI PRs**.\n\nAgent Gate is a deterministic CI firewall for AI-generated pull requests.\n\nIt is not an LLM reviewer.\n\nThat distinction matters.\n\nLLM reviewers help with judgment. Agent Gate verifies deterministic merge evidence.\n\nAn LLM reviewer can tell you whether code looks suspicious. Agent Gate checks whether the PR crossed policy boundaries that should be explainable and repeatable in CI.\n\nThe mental model is:\n\nUse your LLM reviewer for judgment.\n\nUse Agent Gate for deterministic merge evidence.\n\nAgent Gate checks questions that should not require an LLM:\n\nThese are not semantic code review questions. They are merge-boundary questions.\n\nImagine an agent is asked to fix an auth session bug.\n\nThe expected scope might be:\n\n```\nallowed_paths:\n  - src/auth/**\n  - tests/auth/**\n```\n\nBut the PR also changes:\n\n```\nsrc/payments/webhook.ts\n.github/workflows/release.yml\n.mcp.json\n```\n\nA reviewer might catch that. An LLM reviewer might catch that. But I do not want this to depend only on someone noticing.\n\nI want CI to say:\n\n```\nThis PR crossed its declared scope.\nThis PR changed workflow permissions.\nThis PR changed the agent tool surface.\nThis PR needs human decision before merge.\n```\n\nThat is the shape of Agent Gate.\n\nThe current `v0.1.2`\n\nrelease is intentionally focused on deterministic checks.\n\nIt can flag or block the following, depending on policy mode.\n\nAgent Gate can parse a small PR body contract:\n\n```\n<!-- agent-gate-contract\nversion: 1\nagent: codex\ntask: update auth session handling\nallowed_paths:\n  - src/auth/**\n  - tests/auth/**\nrequired_evidence:\n  - matching auth tests changed\n-->\n```\n\nIf the PR changes files outside `allowed_paths`\n\n, Agent Gate reports that as a contract escape.\n\nGitHub Actions workflows are powerful. If a PR changes this:\n\n```\npermissions:\n  contents: read\n```\n\nto this:\n\n```\npermissions:\n  contents: write\n```\n\nthat should be visible before merge.\n\nAgent Gate checks workflow-level permission escalation and dangerous workflow patterns such as:\n\n`write-all`\n\n`id-token: write`\n\n`pull_request_target`\n\nchecking out PR head`secrets.*`\n\nusageFiles like these can change how future agents behave:\n\n```\nAGENTS.md\nCLAUDE.md\n.cursor/**\n.github/copilot-instructions.md\n.mcp.json\n```\n\nA PR that changes `.mcp.json`\n\nis not just changing config. It may be changing which tools an agent can call.\n\nAgent Gate treats those files as an agent control plane and reports drift.\n\nAgent Gate can define high-risk paths:\n\n```\nhigh_risk_paths:\n  auth:\n    paths:\n      - src/auth/**\n    require_tests:\n      - tests/auth/**\n    severity: error\n```\n\nIf auth code changes but no matching auth test file changes, Agent Gate reports missing test evidence.\n\nThis does not prove semantic test coverage. It only checks deterministic file-pattern evidence.\n\nThat limitation is intentional.\n\nOne piece of early feedback was that the report should not start with a wall of rule IDs.\n\nIt should answer the maintainer's first question:\n\nWhat should I do with this PR?\n\nSo the Markdown report now leads with a human decision.\n\nExample shape:\n\n```\nAgent Gate: NEEDS HUMAN DECISION\n\nWhy:\nThis PR changed `.github/workflows/release.yml` and added `secrets.*` usage.\n\nRecommended next step:\nReview the workflow change before merging.\n\nPolicy status:\nWarning today; eligible to become a merge gate after tuning.\n```\n\nThe detailed rule findings still appear underneath.\n\nThe machine-readable JSON decision remains simple:\n\n```\n{\n  \"decision\": \"warn\"\n}\n```\n\nThe human-facing report can say `NEEDS HUMAN DECISION`\n\n, while the machine-readable result stays `pass`\n\n, `warn`\n\n, or `block`\n\n.\n\nAgent Gate is designed around a conservative trust boundary.\n\nAt runtime, the GitHub Action:\n\nIt reads PR metadata and changed-file contents through GitHub APIs.\n\nIt loads `agent-gate.yml`\n\nfrom the PR base branch, not from the untrusted PR branch.\n\nThat matters because a PR should not be able to weaken its own policy.\n\nAgent Gate is available on GitHub Marketplace:\n\n[https://github.com/marketplace/actions/agent-gate-for-ai-prs](https://github.com/marketplace/actions/agent-gate-for-ai-prs)\n\nA minimal workflow looks like this:\n\n```\nname: Agent Gate\n\non:\n  pull_request:\n    types:\n      - opened\n      - synchronize\n      - reopened\n      - edited\n      - labeled\n      - unlabeled\n      - ready_for_review\n\npermissions:\n  contents: read\n  pull-requests: read\n\njobs:\n  agent-gate:\n    runs-on: ubuntu-latest\n    steps:\n      - uses: sjh9714/Agent-Gate@v0.1.2\n        with:\n          github-token: ${{ secrets.GITHUB_TOKEN }}\n          mode: warn\n          fail-on-block: false\n```\n\nI recommend starting with:\n\n```\nmode: warn\nfail-on-block: false\n```\n\nThat gives you an observe path.\n\nFirst learn what Agent Gate finds in your repository. Then promote only the policies that are useful and low-noise into merge gates.\n\nA small starting policy could be:\n\n```\nversion: 1\nmode: warn\n\ncontract:\n  required_for:\n    - agent\n  allow_missing_in_observe_mode: true\n\nagent_detection:\n  labels:\n    - ai\n    - agent\n    - codex\n  branch_patterns:\n    - \"codex/**\"\n    - \"ai/**\"\n\nhigh_risk_paths:\n  workflows:\n    paths:\n      - \".github/workflows/**\"\n    severity: error\n```\n\nThe repository includes an `unsafe-pr-zoo`\n\nwith deterministic fixtures.\n\nAfter cloning the repo and installing dependencies, you can run:\n\n```\npnpm install\npnpm --filter agent-gate build\nnode packages/cli/dist/main.js replay fixtures/unsafe-pr-zoo/workflow-permission-escalation\n```\n\nExample output:\n\n```\nAgent Gate: BLOCKED\n\nERROR workflow/permission-escalation\ncontents permission increased from read to write.\nPath: .github/workflows/release.yml\n\nERROR workflow/dangerous-pattern\n.github/workflows/release.yml contains a dangerous GitHub Actions workflow pattern.\nPath: .github/workflows/release.yml\n```\n\nOther fixtures cover:\n\nAgent Gate is not a replacement for code review.\n\nIt does not try to find every semantic bug.\n\nIt does not know whether a function is logically correct.\n\nIt does not prove that tests are sufficient.\n\nIt does not replace a human reviewer, an LLM reviewer, or normal CI.\n\nIt answers a narrower question:\n\nDid this PR cross deterministic policy boundaries before merge?\n\nThat is the problem I want it to solve well.\n\n`v0.1.2`\n\nis still a prerelease.\n\nKnown limitations:\n\n`issues: write`\n\nand can warn on fork PRs with read-only tokens.I am especially interested in feedback from people trying AI-generated PRs in real repositories.\n\nThe main questions:\n\nFeedback issue:\n\n[https://github.com/sjh9714/Agent-Gate/issues/27](https://github.com/sjh9714/Agent-Gate/issues/27)\n\nRepository:\n\n[https://github.com/sjh9714/Agent-Gate](https://github.com/sjh9714/Agent-Gate)\n\nLLM reviewers are useful.\n\nBut if AI-generated PRs become part of normal engineering workflows, teams will also need deterministic gates.\n\nNot every review question should be probabilistic.\n\nSome questions are simple:\n\nDid this PR stay within scope?\n\nDid workflow permissions escalate?\n\nDid agent control-plane files change?\n\nIs there matching test evidence?\n\nThat is the space Agent Gate is trying to explore.\n\nUse your LLM reviewer for judgment.\n\nUse Agent Gate for deterministic merge evidence.\n\nDisclosure: I used AI assistance to help draft and edit this article, and I reviewed the technical claims before publishing.", "url": "https://wpnews.pro/news/i-built-a-deterministic-ci-firewall-for-ai-generated-pull-requests", "canonical_source": "https://dev.to/sjh9714/i-built-a-deterministic-ci-firewall-for-ai-generated-pull-requests-4o3c", "published_at": "2026-06-15 15:22:57+00:00", "updated_at": "2026-06-15 15:36:32.406085+00:00", "lang": "en", "topics": ["developer-tools", "ai-agents", "large-language-models", "ai-safety"], "entities": ["Agent Gate", "GitHub Actions", "Codex", "LLM"], "alternates": {"html": "https://wpnews.pro/news/i-built-a-deterministic-ci-firewall-for-ai-generated-pull-requests", "markdown": "https://wpnews.pro/news/i-built-a-deterministic-ci-firewall-for-ai-generated-pull-requests.md", "text": "https://wpnews.pro/news/i-built-a-deterministic-ci-firewall-for-ai-generated-pull-requests.txt", "jsonld": "https://wpnews.pro/news/i-built-a-deterministic-ci-firewall-for-ai-generated-pull-requests.jsonld"}}