{"slug": "an-ai-assisted-code-review-pipeline-that-catches-what-humans-skim-past", "title": "An AI-Assisted Code Review Pipeline That Catches What Humans Skim Past", "summary": "A developer has built an AI-assisted code review pipeline that layers deterministic tools before an LLM reviewer to catch semantic bugs humans skim past. The pipeline, which costs cents per pull request, uses formatters, linters, and security scanners as blocking gates, then runs an LLM only on remaining semantic issues. The developer emphasizes that scoping the LLM to intent-level problems prevents noise and keeps human reviewers focused on judgment calls.", "body_md": "Human reviewers are excellent at judgment — is this the right abstraction, does this belong here, will the next person understand it — and terrible at attention. By line 400 of a diff, everyone is skimming. The bugs that ship are almost never the clever ones; they're the swallowed exception, the missing `await`\n\n, the loop that queries the database once per row. A good AI review pipeline is not a replacement for human judgment. It's a way to hand the *attention* work to something that never gets tired, so humans spend their review budget on the parts that actually need a brain.\n\nThis is a walkthrough of a pipeline I run on my own projects: deterministic gates first, an LLM reviewer second, and a hard rule that the LLM only comments where a machine can't. As of mid-2026 the model APIs are cheap and fast enough that this costs cents per pull request, but the design matters more than the model — a badly scoped LLM reviewer produces so much noise that people mute it within a week.\n\nBecause it will comment on everything, and a reviewer that comments on everything gets ignored. The failure mode is well documented on every team that tries it: the bot leaves fourteen comments, twelve are style nits already handled by a formatter, one is a hallucinated \"possible null dereference\" on code that can't be null, and the one real bug is buried in the middle where nobody reads it.\n\nThe fix is layering. Anything a deterministic tool can decide, a deterministic tool *should* decide — it's faster, free, and never wrong about its own rules. The LLM only runs on what's left: the semantic, cross-file, \"this looks right but isn't\" category that linters structurally cannot see.\n\n| Layer | Catches | Tool type | Runs when |\n|---|---|---|---|\n| Formatting | Indentation, import order, quote style | Formatter (Prettier, Black, gofmt) | Pre-commit + CI |\n| Static analysis | Unused vars, obvious type errors, dead code | Linter / type checker | CI, blocking |\n| Security patterns | Hardcoded secrets, known-vuln deps, injection sinks | SAST + secret scanner | CI, blocking |\n| Semantic review | Swallowed errors, missing await, N+1, logic that contradicts the PR description | LLM reviewer | CI, non-blocking comments |\n\nThe takeaway: the LLM is the *last* layer, not the first — it should never see a problem a linter would have caught.\n\nThese are the categories I've measured the LLM layer catching most often, all of which are technically visible in the diff but easy to slide over:\n\n`catch (e) {}`\n\nor `except: pass`\n\n, or a caught error that's logged and then execution continues as if nothing happened.`await`\n\n`>=`\n\nvs `>`\n\n, or an `offset`\n\nthat drops or duplicates a row at page boundaries.That last one is where an LLM genuinely outperforms a linter, because it can read the PR title and description as context. A linter has no idea what the change was *supposed* to do.\n\nThe takeaway: scope the LLM to semantic and intent-level issues, and it stops competing with your linter and starts adding something new.\n\nStart with the deterministic gates as blocking CI steps. Here's the shape of it in GitHub Actions — the review job only runs after the cheap checks pass:\n\n```\nname: review\non: pull_request\n\njobs:\n  gates:\n    runs-on: ubuntu-latest\n    steps:\n      - uses: actions/checkout@v4\n      - run: npm ci\n      - run: npm run lint        # blocking\n      - run: npm run typecheck    # blocking\n      - run: npx secretlint \"**/*\" # blocking\n\n  ai-review:\n    needs: gates          # only spend tokens on diffs that already passed\n    runs-on: ubuntu-latest\n    steps:\n      - uses: actions/checkout@v4\n        with: { fetch-depth: 0 }\n      - run: node scripts/ai-review.mjs\n        env:\n          MODEL_API_KEY: ${{ secrets.MODEL_API_KEY }}\n          PR_NUMBER: ${{ github.event.pull_request.number }}\n          GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}\n```\n\nThe `needs: gates`\n\nline is the whole trick for cost control: you never send a diff to the model until it's already clean by every cheap measure. The review script pulls the diff, sends it with a tightly scoped prompt, and posts findings as review comments:\n\n``` js\n// scripts/ai-review.mjs (sketch)\nimport { execSync } from \"node:child_process\";\n\nconst diff = execSync(\n  `git diff origin/${process.env.GITHUB_BASE_REF}...HEAD`\n).toString();\n\nconst system = `You are a code reviewer. Report ONLY:\n- swallowed or ignored errors\n- missing await / unhandled promises\n- database queries inside loops (N+1)\n- off-by-one errors in slicing or pagination\n- changes that contradict the PR description\n\nDo NOT comment on style, formatting, naming, or anything a linter\nhandles. If you find nothing in these categories, return an empty list.\nFor each finding return: file, line, one-sentence reason, suggested fix.\nReturn JSON only.`;\n\n// send { system, diff, prDescription } to your model API,\n// parse the JSON, and post each finding via the GitHub review API.\n```\n\nThe two load-bearing instructions are \"report ONLY these categories\" and \"if you find nothing, return an empty list.\" Without the second one, models feel obligated to say *something*, and that something is noise. Give the model explicit permission to stay quiet.\n\nThe takeaway: the prompt's job is to make silence the default and comments the exception.\n\nMake the AI layer **non-blocking** and treat its false-positive rate as a metric you actively manage. Two habits keep it honest. First, keep the finding categories in a version-controlled prompt file, and when the bot posts a bad comment, tighten the prompt in the same PR that fixes the underlying issue — the prompt is code and deserves the same review. Second, let humans resolve the bot's comments freely; if a category produces mostly-dismissed comments over a month, cut it from the prompt.\n\nIf you want a managed version of this rather than a hand-rolled script, GitHub's own Copilot code review and third-party bots like CodeRabbit will run an LLM reviewer on every PR without you maintaining the plumbing — the tradeoff is less control over exactly which categories they comment on. The honest drawback of the roll-your-own approach in this post is the opposite: you own the false-positive tuning forever, and that's real ongoing work, not a one-time setup.\n\nThe takeaway: a non-blocking AI reviewer whose prompt you actively prune stays useful; a blocking one whose noise you tolerate gets muted.\n\n**Can AI code review replace human reviewers?**\n\nNo. It replaces the *attention* part of review — catching mechanical bugs a tired human skims past — not the judgment part. Design, architecture, and whether the change should exist at all still need a human, so keep the AI layer non-blocking and advisory.\n\n**How much does an LLM code review pipeline cost to run?**\n\nAs of mid-2026, reviewing a normal-sized pull request costs on the order of cents, because you only send diffs that already passed your linter and type checker. Gating the AI job behind cheap deterministic checks is what keeps the token bill negligible.\n\n**Why does my AI reviewer leave so many useless comments?**\n\nAlmost always because it's the first layer instead of the last, and because the prompt doesn't give it permission to stay silent. Scope it to a short list of semantic categories, tell it to return nothing when it finds nothing, and let a linter handle everything mechanical.\n\nIf you're adding AI to code review, put it last in the chain, not first: formatter, linter, type checker, and secret scanner catch everything mechanical, and the LLM only sees semantic issues those tools structurally can't detect. Make it non-blocking, scope its prompt to a handful of high-value categories, and give it explicit permission to say nothing. Solo developers and small teams get the most out of the hand-rolled script version because they can tune it precisely; teams that don't want to own prompt maintenance are better served by a managed reviewer and accepting less control. The goal is never to review *more* — it's to move human attention off the boring bugs and onto the decisions that actually need a person.", "url": "https://wpnews.pro/news/an-ai-assisted-code-review-pipeline-that-catches-what-humans-skim-past", "canonical_source": "https://dev.to/libme/an-ai-assisted-code-review-pipeline-that-catches-what-humans-skim-past-5hc0", "published_at": "2026-08-10 21:47:08+00:00", "updated_at": "2026-08-10 22:17:17.646149+00:00", "lang": "en", "topics": ["artificial-intelligence", "large-language-models", "developer-tools", "ai-products", "mlops"], "entities": ["GitHub Actions", "Prettier", "Black", "gofmt"], "alternates": {"html": "https://wpnews.pro/news/an-ai-assisted-code-review-pipeline-that-catches-what-humans-skim-past", "markdown": "https://wpnews.pro/news/an-ai-assisted-code-review-pipeline-that-catches-what-humans-skim-past.md", "text": "https://wpnews.pro/news/an-ai-assisted-code-review-pipeline-that-catches-what-humans-skim-past.txt", "jsonld": "https://wpnews.pro/news/an-ai-assisted-code-review-pipeline-that-catches-what-humans-skim-past.jsonld"}}