I built a deterministic CI firewall for AI-generated pull requests

A developer built Agent Gate, a deterministic CI firewall for AI-generated pull requests that verifies merge evidence without relying on LLM judgment. The tool checks policy boundaries such as allowed file paths, workflow permission changes, and agent control plane modifications, producing a human-readable report with a clear decision. Currently in prerelease v0.1.2, Agent Gate aims to make AI-generated PRs easier to inspect before merge.

AI coding agents are getting good enough to open pull requests. That is useful. It also changes the review problem. A normal code review asks: Does this code look correct? An AI-generated PR also raises a different question: Did this agent change something I did not intend, and does this PR have enough evidence to merge? Agent Gate is still a prerelease, so I am starting with a narrow goal: make AI-generated PRs easier to inspect before merge. That second question is why I built Agent Gate for AI PRs . Agent Gate is a deterministic CI firewall for AI-generated pull requests. It is not an LLM reviewer. That distinction matters. LLM reviewers help with judgment. Agent Gate verifies deterministic merge evidence. An LLM reviewer can tell you whether code looks suspicious. Agent Gate checks whether the PR crossed policy boundaries that should be explainable and repeatable in CI. The mental model is: Use your LLM reviewer for judgment. Use Agent Gate for deterministic merge evidence. Agent Gate checks questions that should not require an LLM: These are not semantic code review questions. They are merge-boundary questions. Imagine an agent is asked to fix an auth session bug. The expected scope might be: allowed paths: - src/auth/ - tests/auth/ But the PR also changes: src/payments/webhook.ts .github/workflows/release.yml .mcp.json A reviewer might catch that. An LLM reviewer might catch that. But I do not want this to depend only on someone noticing. I want CI to say: This PR crossed its declared scope. This PR changed workflow permissions. This PR changed the agent tool surface. This PR needs human decision before merge. That is the shape of Agent Gate. The current v0.1.2 release is intentionally focused on deterministic checks. It can flag or block the following, depending on policy mode. Agent Gate can parse a small PR body contract: < -- agent-gate-contract version: 1 agent: codex task: update auth session handling allowed paths: - src/auth/ - tests/auth/ required evidence: - matching auth tests changed -- If the PR changes files outside allowed paths , Agent Gate reports that as a contract escape. GitHub Actions workflows are powerful. If a PR changes this: permissions: contents: read to this: permissions: contents: write that should be visible before merge. Agent Gate checks workflow-level permission escalation and dangerous workflow patterns such as: write-all id-token: write pull request target checking out PR head secrets. usageFiles like these can change how future agents behave: AGENTS.md CLAUDE.md .cursor/ .github/copilot-instructions.md .mcp.json A PR that changes .mcp.json is not just changing config. It may be changing which tools an agent can call. Agent Gate treats those files as an agent control plane and reports drift. Agent Gate can define high-risk paths: high risk paths: auth: paths: - src/auth/ require tests: - tests/auth/ severity: error If auth code changes but no matching auth test file changes, Agent Gate reports missing test evidence. This does not prove semantic test coverage. It only checks deterministic file-pattern evidence. That limitation is intentional. One piece of early feedback was that the report should not start with a wall of rule IDs. It should answer the maintainer's first question: What should I do with this PR? So the Markdown report now leads with a human decision. Example shape: Agent Gate: NEEDS HUMAN DECISION Why: This PR changed .github/workflows/release.yml and added secrets. usage. Recommended next step: Review the workflow change before merging. Policy status: Warning today; eligible to become a merge gate after tuning. The detailed rule findings still appear underneath. The machine-readable JSON decision remains simple: { "decision": "warn" } The human-facing report can say NEEDS HUMAN DECISION , while the machine-readable result stays pass , warn , or block . Agent Gate is designed around a conservative trust boundary. At runtime, the GitHub Action: It reads PR metadata and changed-file contents through GitHub APIs. It loads agent-gate.yml from the PR base branch, not from the untrusted PR branch. That matters because a PR should not be able to weaken its own policy. Agent Gate is available on GitHub Marketplace: https://github.com/marketplace/actions/agent-gate-for-ai-prs https://github.com/marketplace/actions/agent-gate-for-ai-prs A minimal workflow looks like this: name: Agent Gate on: pull request: types: - opened - synchronize - reopened - edited - labeled - unlabeled - ready for review permissions: contents: read pull-requests: read jobs: agent-gate: runs-on: ubuntu-latest steps: - uses: sjh9714/Agent-Gate@v0.1.2 with: github-token: ${{ secrets.GITHUB TOKEN }} mode: warn fail-on-block: false I recommend starting with: mode: warn fail-on-block: false That gives you an observe path. First learn what Agent Gate finds in your repository. Then promote only the policies that are useful and low-noise into merge gates. A small starting policy could be: version: 1 mode: warn contract: required for: - agent allow missing in observe mode: true agent detection: labels: - ai - agent - codex branch patterns: - "codex/ " - "ai/ " high risk paths: workflows: paths: - ".github/workflows/ " severity: error The repository includes an unsafe-pr-zoo with deterministic fixtures. After cloning the repo and installing dependencies, you can run: pnpm install pnpm --filter agent-gate build node packages/cli/dist/main.js replay fixtures/unsafe-pr-zoo/workflow-permission-escalation Example output: Agent Gate: BLOCKED ERROR workflow/permission-escalation contents permission increased from read to write. Path: .github/workflows/release.yml ERROR workflow/dangerous-pattern .github/workflows/release.yml contains a dangerous GitHub Actions workflow pattern. Path: .github/workflows/release.yml Other fixtures cover: Agent Gate is not a replacement for code review. It does not try to find every semantic bug. It does not know whether a function is logically correct. It does not prove that tests are sufficient. It does not replace a human reviewer, an LLM reviewer, or normal CI. It answers a narrower question: Did this PR cross deterministic policy boundaries before merge? That is the problem I want it to solve well. v0.1.2 is still a prerelease. Known limitations: issues: write and can warn on fork PRs with read-only tokens.I am especially interested in feedback from people trying AI-generated PRs in real repositories. The main questions: Feedback issue: https://github.com/sjh9714/Agent-Gate/issues/27 https://github.com/sjh9714/Agent-Gate/issues/27 Repository: https://github.com/sjh9714/Agent-Gate https://github.com/sjh9714/Agent-Gate LLM reviewers are useful. But if AI-generated PRs become part of normal engineering workflows, teams will also need deterministic gates. Not every review question should be probabilistic. Some questions are simple: Did this PR stay within scope? Did workflow permissions escalate? Did agent control-plane files change? Is there matching test evidence? That is the space Agent Gate is trying to explore. Use your LLM reviewer for judgment. Use Agent Gate for deterministic merge evidence. Disclosure: I used AI assistance to help draft and edit this article, and I reviewed the technical claims before publishing.