AI code review in pull requests works when it is treated as a diff-scoped filter with a hard noise budget, not as a reviewer with opinions. The bot should catch the class of defects that deterministic tools miss and that humans skim past: a swallowed error, a missing authorization check, a test that asserts nothing, a retry that will double-charge. It should not comment on naming, architecture taste, or anything a linter already covers. Teams that get this right see review time fall and defect escapes drop. Teams that point a model at the whole repository get a comment firehose, reviewers learn to dismiss everything, and the automation becomes a net tax.
The numbers from the research community are useful here because they set expectations honestly. Meta's TestGen-LLM work, published at FSE 2024 and available as Automated Unit Test Improvement using Large Language Models at Meta, found that 75 percent of generated test cases built correctly, 57 percent passed reliably, and 25 percent increased coverage, with 73 percent of recommendations accepted for production. That is a serious hit rate for generated code, and it is also nowhere near the "replaces your reviewer" claim in most vendor decks.
This guide covers the configuration that produces signal: what to automate, how to gate it, how to stop the spam, how to keep the reviewer itself from becoming an attack surface, and how to measure whether it is helping.
| Finding class | AI reviewer performance | Better handled by |
|---|---|---|
| Missing error handling around a new call | Strong, especially with the diff plus the called function | AI review |
| Authorization check missing on a new route | Strong with a rules file naming the pattern | AI review plus a deterministic auth test |
| Injection and secrets in new code | Moderate, inconsistent | SAST, secret scanning, AI as second pass |
| Logic that contradicts business rules | Weak without domain context | Human review |
| Architecture and module boundaries | Weak and prone to taste arguments | Human review, design docs |
| Formatting, style, naming conventions | Reliable but low value | Linters and formatters, not a model |
| Test quality (assertions that prove nothing) | Moderate to strong | AI review plus mutation testing |
| Performance regressions | Weak at review time | Benchmarks and load tests |
The pattern: the model is good at local, checkable properties in a diff, and poor at anything that requires knowledge it was not given. Configure it for the first column and stop trying to make it the third.
Review attention is a finite resource and a bot can exhaust it faster than any human. The GitClear research on AI-assisted code quality reports a striking shift in what codebases contain: refactoring is down roughly 70 percent while code duplication is up about 81 percent across the period studied, documented in their AI code quality research. That is exactly the environment where a noisy reviewer does harm: more duplicated blocks means more near-identical findings, which means more comments, which means dismissal becomes reflex.
DORA's 2024 Accelerate State of DevOps report adds the organisational frame. AI adoption showed measurable productivity gains alongside a decrease in delivery stability and throughput at the team level. Read together with GitClear, the implication is not "avoid AI" but "AI raises output volume, and review capacity has to absorb it." An uncalibrated review bot makes that harder, not easier.
So set a budget before you enable anything: at most three comments per pull request, one blocking severity, zero style comments. If the bot cannot fit its findings into that budget, it stays advisory until it can.
The configuration that works is narrow, explicit, and diff-scoped.
pull_request with opened, synchronize, and reopened, which is the documented pattern in GitHub's .github/ai-review-rules.md) listing what must be flagged, what must be ignored, and the severity definitions. This is the single most important configuration decision. Without it, every reviewer gets a different opinion per run.pull_request_target with a checkout of the fork's head and a secret in the environment; that is the documented recipe for remote code execution through a workflow.
Two configuration details matter more than the model you pick. First, the rules file turns "review my code" into a checkable spec, which is what makes comments consistent enough to trust. Second, per-hunk deduplication is the difference between a bot that feels useful and a bot that feels like a repeating alarm.
Spam is the default state of an uncalibrated reviewer, so treat noise control as a feature with its own engineering.
ai-review: ignore annotations with a reason and a review date, and report on how many expirations are outstanding. Permanent suppression is how teams end up with a bot that says nothing and a policy nobody remembers.
If you want a comparison of which tools do this well and what they cost, our AI code review tools review covers the vendor side. This article is about the workflow around whichever tool you pick.
| Check | Automated (deterministic) | AI review | Human |
|---|---|---|---|
| Formatting, lint, type errors | Yes | No | No |
| Unit test pass, build, coverage floor | Yes | No | No |
| Secret scanning, dependency CVEs | Yes | No | Escalation only |
| Authorization pattern on new routes | Partial (policy tests) | Yes | On failures |
| Error handling on new external calls | No | Yes | Spot check |
| Business rule correctness | No | No | Yes |
| Data migration and rollback safety | Partial (dry run) | Yes | Yes |
| Architecture and API design | No | Draft only | Yes |
The rule that keeps this honest: never let an AI comment be the only gate on a merge. It is an additional signal, and the branch protection rules should not be weakened because "the bot looked at it." Google's code review guidance says the reviewer responds quickly and the change stays small; the reviewer's authority is not delegated to a bot.
The sequence that keeps review latency low:
The important property is ordering under time pressure. When the human review starts before the AI finishes, reviewers learn to ignore late bot comments, which is the same failure as spam by a different route. Start the AI job first.
A minimal workflow that gates on deterministic checks, then runs a diff-scoped review, then posts at most three findings:
name: pr-review
on:
pull_request:
types: [opened, synchronize, reopened]
permissions:
contents: read
pull-requests: write
jobs:
gates:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
- run: npm ci
- run: npm run lint
- run: npm run typecheck
- run: npm test -- --run
ai-review:
needs: gates
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
with:
fetch-depth: 0
- uses: actions/setup-node@v4
with:
node-version: 22
- run: node scripts/ai-review.mjs
env:
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
MODEL_API_KEY: ${{ secrets.MODEL_API_KEY }}
MAX_COMMENTS: "3"
BLOCKING_SEVERITIES: "critical"
And the reviewer script's contract, which matters more than its internals:
// 1. collect the diff and only the enclosing functions
// 2. load .github/ai-review-rules.md and pass it as policy
// 3. request structured JSON: [{file, line, severity, title, reason, fix}]
// 4. drop anything below the configured severity
// 5. dedupe by (file, hunk, title) against previous runs
// 6. post at most MAX_COMMENTS inline, the rest as one summary
// 7. exit non-zero only when a blocking severity is present
Ask for structured output rather than prose. Our structured outputs guide covers the schema pattern; validating the reviewer's JSON is what makes comment caps and severity gates possible at all.
This is the part most teams miss. The content being reviewed is untrusted input: a pull request can come from a fork, from a compromised contributor account, or from a dependency bot. A model that reads a diff and follows instructions can be manipulated by comments embedded in the code.
Concrete rules:
The failure mode to plan for is indirect prompt injection inside a pull request, which is the same class we cover in our prompt injection defense guide. A bot that reviews untrusted diffs and has any tool access is a text channel into your CI.
Stage 1, shadow mode. Run the reviewer on all PRs but post nothing. Compare its findings against human review for two weeks. The output is a false-positive estimate and a draft rules file.
Stage 2, advisory. Post comments, block nothing. Measure dismissal rate and comment volume per PR. If dismissal exceeds roughly one in three findings, the rules file needs work before anything blocks.
Stage 3, selective blocking. Enable blocking for the small allowlist of severe findings only, on the repositories where the false-positive rate is measured and low.
Stage 4, calibration. Monthly: review dismissals, re-run a labeled set of historical PRs, check for model drift after vendor upgrades. This is the same discipline as any evaluation pipeline, which our LLM output drift guide covers for model-facing systems.
| Metric | What good looks like | What it catches |
|---|---|---|
| Review latency (first human comment) | Falling or stable | Bot slowing the human path |
| Comment acceptance rate | Above 60 to 70 percent | Spam, miscalibrated rules |
| Dismissal reasons top 3 | Stable and shrinking | Rules debt |
| Defect escapes per 100 PRs | Falling | Whether the reviewer adds real value |
| Comments per PR | Under 3 | Noise budget breach |
| Blocking findings per week | Single digits | Overblocking |
Avoid measuring "findings produced." A model can produce findings forever and that number only proves the API works. What matters is accepted findings that led to a change, and defects that stopped reaching production.
AI code review earns its place as a narrow, diff-scoped filter with a hard noise budget, deterministic gates in front of it, and a human reviewer who keeps authority. The research supports a real but bounded contribution: generated tests that pass and get accepted at meaningful rates, and local defect detection that humans skim past. It does not support replacing review judgement or letting a bot's silence mean approval.
Configure the rules file first, enforce the comment budget in code, keep the reviewer out of your secrets and write paths, roll it out in shadow mode, and measure accepted findings and escaped defects rather than volume. Do that and review gets faster without getting weaker.
If you want a second set of eyes on the workflow before the bot starts blocking merges, our software development practice and AI consulting both review CI and review automation for small teams.