An AI-Assisted Code Review Pipeline That Catches What Humans Skim Past

Human reviewers are excellent at judgment — is this the right abstraction, does this belong here, will the next person understand it — and terrible at attention. By line 400 of a diff, everyone is skimming. The bugs that ship are almost never the clever ones; they're the swallowed exception, the missing await

, the loop that queries the database once per row. A good AI review pipeline is not a replacement for human judgment. It's a way to hand the attention work to something that never gets tired, so humans spend their review budget on the parts that actually need a brain.

This is a walkthrough of a pipeline I run on my own projects: deterministic gates first, an LLM reviewer second, and a hard rule that the LLM only comments where a machine can't. As of mid-2026 the model APIs are cheap and fast enough that this costs cents per pull request, but the design matters more than the model — a badly scoped LLM reviewer produces so much noise that people mute it within a week.

Because it will comment on everything, and a reviewer that comments on everything gets ignored. The failure mode is well documented on every team that tries it: the bot leaves fourteen comments, twelve are style nits already handled by a formatter, one is a hallucinated "possible null dereference" on code that can't be null, and the one real bug is buried in the middle where nobody reads it.

The fix is layering. Anything a deterministic tool can decide, a deterministic tool should decide — it's faster, free, and never wrong about its own rules. The LLM only runs on what's left: the semantic, cross-file, "this looks right but isn't" category that linters structurally cannot see.

The takeaway: the LLM is the last layer, not the first — it should never see a problem a linter would have caught.

These are the categories I've measured the LLM layer catching most often, all of which are technically visible in the diff but easy to slide over:

catch (e) {}

or except: pass

, or a caught error that's logged and then execution continues as if nothing happened.await

>=

vs >

, or an offset

that drops or duplicates a row at page boundaries.That last one is where an LLM genuinely outperforms a linter, because it can read the PR title and description as context. A linter has no idea what the change was supposed to do.

The takeaway: scope the LLM to semantic and intent-level issues, and it stops competing with your linter and starts adding something new.

Start with the deterministic gates as blocking CI steps. Here's the shape of it in GitHub Actions — the review job only runs after the cheap checks pass:

name: review
on: pull_request

jobs:
  gates:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - run: npm ci
      - run: npm run lint        # blocking
      - run: npm run typecheck    # blocking
      - run: npx secretlint "**/*" # blocking

  ai-review:
    needs: gates          # only spend tokens on diffs that already passed
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
        with: { fetch-depth: 0 }
      - run: node scripts/ai-review.mjs
        env:
          MODEL_API_KEY: ${{ secrets.MODEL_API_KEY }}
          PR_NUMBER: ${{ github.event.pull_request.number }}
          GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}

The needs: gates

line is the whole trick for cost control: you never send a diff to the model until it's already clean by every cheap measure. The review script pulls the diff, sends it with a tightly scoped prompt, and posts findings as review comments:

// scripts/ai-review.mjs (sketch)
import { execSync } from "node:child_process";

const diff = execSync(
  `git diff origin/${process.env.GITHUB_BASE_REF}...HEAD`
).toString();

const system = `You are a code reviewer. Report ONLY:
- swallowed or ignored errors
- missing await / unhandled promises
- database queries inside loops (N+1)
- off-by-one errors in slicing or pagination
- changes that contradict the PR description

Do NOT comment on style, formatting, naming, or anything a linter
handles. If you find nothing in these categories, return an empty list.
For each finding return: file, line, one-sentence reason, suggested fix.
Return JSON only.`;

// send { system, diff, prDescription } to your model API,
// parse the JSON, and post each finding via the GitHub review API.

The two load-bearing instructions are "report ONLY these categories" and "if you find nothing, return an empty list." Without the second one, models feel obligated to say something, and that something is noise. Give the model explicit permission to stay quiet.

The takeaway: the prompt's job is to make silence the default and comments the exception.

Make the AI layer non-blocking and treat its false-positive rate as a metric you actively manage. Two habits keep it honest. First, keep the finding categories in a version-controlled prompt file, and when the bot posts a bad comment, tighten the prompt in the same PR that fixes the underlying issue — the prompt is code and deserves the same review. Second, let humans resolve the bot's comments freely; if a category produces mostly-dismissed comments over a month, cut it from the prompt.

If you want a managed version of this rather than a hand-rolled script, GitHub's own Copilot code review and third-party bots like CodeRabbit will run an LLM reviewer on every PR without you maintaining the plumbing — the tradeoff is less control over exactly which categories they comment on. The honest drawback of the roll-your-own approach in this post is the opposite: you own the false-positive tuning forever, and that's real ongoing work, not a one-time setup.

The takeaway: a non-blocking AI reviewer whose prompt you actively prune stays useful; a blocking one whose noise you tolerate gets muted.

Can AI code review replace human reviewers?

No. It replaces the attention part of review — catching mechanical bugs a tired human skims past — not the judgment part. Design, architecture, and whether the change should exist at all still need a human, so keep the AI layer non-blocking and advisory.

How much does an LLM code review pipeline cost to run?

As of mid-2026, reviewing a normal-sized pull request costs on the order of cents, because you only send diffs that already passed your linter and type checker. Gating the AI job behind cheap deterministic checks is what keeps the token bill negligible.

Why does my AI reviewer leave so many useless comments?

Almost always because it's the first layer instead of the last, and because the prompt doesn't give it permission to stay silent. Scope it to a short list of semantic categories, tell it to return nothing when it finds nothing, and let a linter handle everything mechanical.

If you're adding AI to code review, put it last in the chain, not first: formatter, linter, type checker, and secret scanner catch everything mechanical, and the LLM only sees semantic issues those tools structurally can't detect. Make it non-blocking, scope its prompt to a handful of high-value categories, and give it explicit permission to say nothing. Solo developers and small teams get the most out of the hand-rolled script version because they can tune it precisely; teams that don't want to own prompt maintenance are better served by a managed reviewer and accepting less control. The goal is never to review more — it's to move human attention off the boring bugs and onto the decisions that actually need a person.