AI Code Review at Scale: How We Use Claude to Review Every PR Before Humans See It

A software team has deployed an AI first-pass reviewer using Anthropic's Claude in their GitHub Actions pipeline, which reviews every pull request within 90 seconds and catches 30–40% of real issues before human review, processing 70–100 PRs weekly and 4,200 PRs over six months. The system chunks diffs by file and posts inline comments, allowing senior engineers to focus on architectural and business logic issues instead of spending 4–6 hours daily on routine checks.

AI Code Review at Scale: How We Use Claude to Review Every PR Before Humans See It Our team reviews 70–100 pull requests a week. Before we automated the first pass, a senior engineer was spending 4–6 hours a day on reviews that ranged from genuinely interesting architectural decisions to "you forgot to close the stream." We weren't going to hire our way out of that — the solution Our team reviews 70–100 pull requests a week. Before we automated the first pass, a senior engineer was spending 4–6 hours a day on reviews that ranged from genuinely interesting architectural decisions to "you forgot to close the stream." We weren't going to hire our way out of that — the solution was changing what humans review, not adding more humans. Six months ago we shipped an AI first-pass reviewer into our GitHub Actions pipeline. It runs on every PR, posts inline comments on the diff within 90 seconds of the push, and consistently catches 30–40% of real issues before a human reads a single line. Here's the complete architecture, prompt, workflow, and the numbers after 4,200 pull requests. AI code review is not a replacement for human review. It's a filter and an amplifier. The AI catches: null pointer patterns, missing null checks, obvious resource leaks, SQL injection risks, hardcoded secrets, synchronization problems, missing error handling, type safety issues, test coverage gaps in the changed code. Humans catch: architectural tradeoffs, business logic correctness, team convention alignment, long-term maintainability decisions, subtle concurrency bugs that require understanding the full system. A well-configured AI reviewer handles the first category reliably enough that human reviewers spend their time almost entirely on the second. That's the value — not replacing judgment, but protecting it. flowchart TD A Developer pushes\nto feature branch -- |webhook| B GitHub Actions\nai-review.yml B -- C Fetch PR diff\nvia GitHub API C -- D{Diff size check} D -- | 1000 lines| E Chunk diff\nby file D -- |<= 1000 lines| F Single request E -- G Review each chunk\nwith Claude API F -- G G -- H Aggregate findings H -- I{Findings exist?} I -- |Yes| J Post inline comments\nvia GitHub PR Review API I -- |No| K Post approval\n'LGTM from AI reviewer' J -- L Human reviewer\nsees flagged lines L -- M Human focuses on\nwhat AI missed style G fill: 1e3a5f,color: 7dd3fc style J fill: 065f46,color: 6ee7b7 style M fill: 374151,color: d1d5db Two design decisions worth explaining: Chunking by file, not by line count. When a PR is large, splitting at an arbitrary line boundary produces incoherent context. Splitting by file means each chunk is a complete, reviewable unit — the AI sees the full before/after for each file. Inline comments, not a summary. A summary paragraph at the bottom of the PR thread gets ignored. An inline comment on line 47 of PaymentService.java is impossible to miss. The GitHub PR Review API lets you attach comments to specific lines in the diff — that's what makes the feedback actionable. .github/workflows/ai-review.yml name: AI Code Review on: pull request: types: opened, synchronize branches: main, develop jobs: review: runs-on: ubuntu-latest Skip dependabot, release branches, and draft PRs if: | github.actor = 'dependabot bot ' && startsWith github.head ref, 'release/' && github.event.pull request.draft == false permissions: pull-requests: write contents: read steps: - uses: actions/checkout@v4 with: fetch-depth: 0 - name: Fetch PR diff id: diff env: GH TOKEN: ${{ secrets.GITHUB TOKEN }} run: | gh pr diff ${{ github.event.pull request.number }} \ --repo ${{ github.repository }} pr.diff lines=$ wc -l < pr.diff echo "lines=$lines" $GITHUB OUTPUT echo "PR diff: $lines lines" - name: AI Review env: ANTHROPIC API KEY: ${{ secrets.ANTHROPIC API KEY }} GH TOKEN: ${{ secrets.GITHUB TOKEN }} PR NUMBER: ${{ github.event.pull request.number }} REPO: ${{ github.repository }} BASE SHA: ${{ github.event.pull request.base.sha }} HEAD SHA: ${{ github.event.pull request.head.sha }} run: | node .github/scripts/ai-review.mjs The workflow is intentionally minimal — all the logic lives in ai-review.mjs so it can be tested locally without triggering Actions. // .github/scripts/ai-review.mjs import { readFileSync } from "node:fs"; import Anthropic from "@anthropic-ai/sdk"; const client = new Anthropic { apiKey: process.env.ANTHROPIC API KEY } ; const GH TOKEN = process.env.GH TOKEN; const PR NUMBER = process.env.PR NUMBER; const REPO = process.env.REPO; const HEAD SHA = process.env.HEAD SHA; const REVIEW PROMPT = You are a senior Java/Spring Boot engineer reviewing a pull request diff. Review the diff below and identify ONLY real, high-confidence issues. Do not flag: - Style preferences or opinions - Minor naming issues that don't affect correctness - Missing Javadoc on non-public methods - Issues already present in unchanged lines context lines starting with space Flag these categories when you find them: - NULL DEREF: Potential NullPointerException or missing null check on added lines - RESOURCE LEAK: Stream, connection, or IO resource opened without try-with-resources - SECURITY: SQL injection, hardcoded secret, XSS vector, unsafe deserialization - CONCURRENCY: Unsynchronized shared mutable state, missing volatile, race condition - ERROR HANDLING: Swallowed exception empty catch block , exception converted to void - LOGIC BUG: Off-by-one, incorrect boolean logic, unreachable code - TEST GAP: New public method with no corresponding test in the diff Respond with a JSON array. Each object has: { "file": "path/to/File.java", "line": <line number in the NEW file where the issue appears , "category": "<one of the categories above ", "severity": "high" | "medium" | "low", "comment": "<specific, actionable comment referencing the exact code — max 3 sentences " } If you find no real issues, respond with: Do not add preamble or explanation outside the JSON array. ; async function reviewChunk diffChunk { const message = await client.messages.create { model: "claude-opus-5", // Use strongest model for code review max tokens: 4096, messages: { role: "user", content: ${REVIEW PROMPT}\n\n\ \ \ diff\n${diffChunk}\n\ \ \ } } ; const text = message.content 0 .text.trim ; try { return JSON.parse text ; } catch { // Model returned non-JSON rare — skip this chunk console.warn "Non-JSON response from model, skipping chunk" ; return ; } } function splitDiffByFile diff { const chunks = ; const lines = diff.split "\n" ; let current = ; let inFile = false; for const line of lines { if line.startsWith "diff --git" { if current.length 0 && inFile chunks.push current.join "\n" ; current = line ; inFile = true; } else { current.push line ; } } if current.length 0 && inFile chunks.push current.join "\n" ; return chunks; } async function postReviewComments findings { if findings.length === 0 { // Post a clean LGTM review await ghApi /repos/${REPO}/pulls/${PR NUMBER}/reviews , { commit id: HEAD SHA, event: "COMMENT", body: " AI Reviewer — No issues found in this diff. Passing to human review.", comments: } ; return; } // Group by severity for the summary body const high = findings.filter f = f.severity === "high" ; const medium = findings.filter f = f.severity === "medium" ; const low = findings.filter f = f.severity === "low" ; const summary = AI Reviewer — Found ${findings.length} potential issue s : , high.length ? 🔴 ${high.length} high severity : "", medium.length ? 🟡 ${medium.length} medium severity : "", low.length ? 🟢 ${low.length} low severity : "", "", " These are automated findings. Use judgment — the AI can be wrong. " .filter Boolean .join "\n" ; // Map findings to inline PR review comments const comments = findings .filter f = f.file && f.line .map f = { path: f.file, position: f.line, // GitHub API: position in the diff, not file line number body: ${f.category} ${severityEmoji f.severity } ${f.comment} } ; await ghApi /repos/${REPO}/pulls/${PR NUMBER}/reviews , { commit id: HEAD SHA, event: "COMMENT", body: summary, comments } ; } function severityEmoji s { return { high: "🔴", medium: "🟡", low: "🟢" } s ?? "⚪"; } async function ghApi path, body { const res = await fetch https://api.github.com${path} , { method: "POST", headers: { Authorization: Bearer ${GH TOKEN} , "Content-Type": "application/json", "X-GitHub-Api-Version": "2022-11-28" }, body: JSON.stringify body } ; if res.ok { const err = await res.text ; throw new Error GitHub API ${path} failed ${res.status} : ${err} ; } } // ── Main ────────────────────────────────────────────────────────────────────── const diff = readFileSync "pr.diff", "utf-8" ; // Skip if no meaningful diff binary files, lock files only, etc. if diff.trim .length < 50 { console.log "Diff too small — skipping AI review" ; process.exit 0 ; } const chunks = splitDiffByFile diff ; const MAX CHUNK CHARS = 12 000; // ~3K tokens — leaves room for prompt + response // Large file diffs get truncated to avoid exceeding model context const safeChunks = chunks.map c = c.length MAX CHUNK CHARS ? c.slice 0, MAX CHUNK CHARS + "\n... truncated " : c ; console.log Reviewing ${safeChunks.length} file chunk s ... ; const allFindings = await Promise.all safeChunks.map reviewChunk .flat ; // Deduplicate by file+line same issue caught in multiple chunks const seen = new Set ; const findings = allFindings.filter f = { const key = ${f.file}:${f.line}:${f.category} ; if seen.has key return false; seen.add key ; return true; } ; console.log Found ${findings.length} issue s after deduplication ; findings.forEach f = console.log ${f.severity} ${f.file}:${f.line} — ${f.category} ; await postReviewComments findings ; console.log "Review posted." ; The prompt is the hardest part. A poorly calibrated prompt produces one of two failures: Too noisy: The AI flags style issues, subjective naming, and things that are obviously fine. Engineers start ignoring the comments within a week. Too quiet: The AI reports nothing on every PR. Zero trust, zero value. Three decisions made our prompt work: Explicit exclusion list. "Do not flag style preferences, minor naming, missing Javadoc on non-public methods." Without this, GPT-4 and Claude both default to acting like a linter. Category-constrained output. Constraining to seven named categories forces the model to be specific and prevents vague findings like "this code could be improved." Every finding must fit a category. Strict JSON-only output. "Respond with a JSON array. If you find no real issues, respond with: " — no preamble, no explanation. This makes the output machine-parseable without fragile regex extraction. NOTE claude-opus-5 for code review is deliberate. Haiku is faster and 20× cheaper but misses subtle concurrency issues and null-deref patterns that require multi-step reasoning. For code review — where you're making a quality-critical decision — use the strongest available model. Generic code review prompts miss Java-specific failure modes. We added these to the category list after observing what slipped through: // Additions to REVIEW PROMPT for Java/Spring codebases: const JAVA EXTENSIONS = Additional Java/Spring-specific checks: - OPTIONAL MISUSE: Calling Optional.get without isPresent check, or using Optional in a field - TRANSACTIONAL RISK: @Transactional on a private method Spring proxy cannot intercept , or @Transactional with noRollbackFor swallowing checked exceptions - JPA PITFALL: N+1 query in a loop entity.getCollection called per iteration without JOIN FETCH , or LazyInitializationException risk accessing lazy collection outside transaction boundary - THREAD POOL STARVATION: Blocking call Thread.sleep, CompletableFuture.get without timeout inside a reactive chain or a thread pool shared with virtual threads ; These four categories alone catch the most expensive bugs in Spring Boot services. After the first week, we tracked every AI comment as either: True positive — the human reviewer confirmed it was a real issue False positive — the human reviewer dismissed it We used this to tune the prompt iteratively. Week 1: 58% true positive rate. Week 6: 81%. The biggest false-positive categories and their fixes: False Positive Type Fix Flagging test code for missing error handling Added "do not flag test files Test.java, Spec.java for missing exception handling" Flagging Spring bean injection as "unused field" Added "fields annotated with @Autowired, @Inject, or @Value are not unused" Flagging intentionally empty catch blocks Added "if the catch block contains a comment, treat it as intentional" Flagging abstract methods for missing implementation Added "do not flag abstract method declarations" Prompt tuning is ongoing. We export false-positive cases monthly and run them against prompt candidates before deploying a new version. flowchart LR subgraph "Before AI Review" A 4.2h avg\nhuman review time\nper engineer/day B 18% of merged PRs\nhad post-merge bug fixes\nwithin 72h C ~45 min\navg time-to-first-review end subgraph "After AI Review" D 2.7h avg\nhuman review time\nper engineer/day E 11% of merged PRs\nhad post-merge bug fixes\nwithin 72h F 90 sec\nfirst AI review\nthen human follows end A -.- |"−36%"| D B -.- |"−39%"| E C -.- |"−98%"| F style D fill: 065f46,color: 6ee7b7 style E fill: 065f46,color: 6ee7b7 style F fill: 065f46,color: 6ee7b7 What actually changed: Senior engineers stopped reviewing trivial null-check issues, stream leaks, and basic error handling gaps — the AI catches those reliably Human review became genuinely architectural — reviewers now spend time on "is this the right abstraction?" not "did you close the connection?" New engineers get faster feedback loops: AI review in 90 seconds means they can iterate on a PR before a human is even available What didn't change: Logic bugs requiring business domain knowledge — AI catch rate here is low ~15% Performance issues requiring profiling — AI never reliably flags these API design decisions — the AI has no opinion worth having on whether your REST endpoint should be POST /documents/search vs GET /documents?q= Cost Breakdown 4,200 PRs, average diff size of ~380 lines across 6 files: Item Volume Cost Input tokens diff + prompt ~2,400 per PR avg Output tokens findings JSON ~400 per PR avg Model claude-opus-5 $15/M input, $75/M output Per PR cost ~$0.066 Monthly 700 PRs ~$46/month $46/month to give every PR a 90-second first pass. For comparison: one hour of a senior engineer's time at market rate costs more than a month of AI reviews. NOTE claude-haiku-4-5 security, null deref, resource leaks only and escalating uncertain findings to Opus cuts cost to ~$8/month at the same volume. We tested this and found a 12% drop in true positive rate — acceptable for some teams, not for ours. Auto-blocking merges on AI findings. We tried this for two weeks. Engineers worked around it by pushing fixes to silence the AI rather than evaluating whether the finding was real. Now AI findings are advisory — they inform the human reviewer but never block the merge. Reviewing the full repository history on every PR. Only review the diff. Reviewing unchanged lines produces false positives at an unusable rate and wastes tokens. One prompt for all languages. Our Python services use a different prompt with different category list. Java/Spring patterns don't map to Python anti-patterns, and a shared prompt produces noise in both. Not filtering bot PRs. Dependabot, Renovate, and automated release PRs don't need AI review. Filter them early or you burn tokens reviewing lock file changes. Posting findings as a separate comment thread. Using the GitHub PR Review API inline comments on the diff is 4× more actionable than a single summary comment at the bottom. Engineers read inline comments; they skim summary paragraphs. Add ANTHROPIC API KEY to your repository secrets Settings → Secrets and variables → Actions Copy .github/workflows/ai-review.yml and .github/scripts/ai-review.mjs to your repository Tune the prompt to your stack. The seven categories above work for Java/Spring. Add language-specific patterns for your codebase see the Java extensions block above . Run in shadow mode for 2 weeks first. Post findings to a private Slack channel instead of the PR. Track true vs false positive rate before exposing it to your engineers. Don't skip this step — a noisy AI reviewer destroys trust instantly. Add a human override label. We honor a skip-ai-review label on PRs for cases where the author knows the diff will confuse the model e.g., large auto-generated files . Respecting human judgment about when to skip builds trust. Add this condition to the workflow job if: | github.actor = 'dependabot bot ' && startsWith github.head ref, 'release/' && github.event.pull request.draft == false && contains github.event.pull request.labels. .name, 'skip-ai-review' AI review is a filter, not a judge. It protects senior engineers' time for architectural decisions by handling correctness checks they shouldn't be doing manually. Inline comments on the diff beat summaries. The GitHub PR Review API is the right tool — a bottom-of-thread summary gets ignored. Prompt calibration is the work. The model is not the variable; the prompt is. Track true vs false positive rate and tune monthly. Use your strongest model. Code review is a quality-critical task — 20× cost savings on a cheaper model are not worth a 15% drop in catch rate. Shadow mode before going live. Two weeks of private feedback before posting to PRs is the difference between engineers trusting the system and ignoring it. Advisory, not blocking. AI findings inform human reviewers; they should never be the sole gate on a merge. The AI is wrong sometimes, and engineers need to feel empowered to disagree. The 36% reduction in human review time is real, but the number that actually changed how our team works is the post-merge bug fix rate dropping from 18% to 11%. That's the metric that matters — fewer bugs reaching production, not faster reviews. The prompt and workflow files are available as starting points — link in bio. If you're running something similar or found categories that work well for your stack, I'd be curious what you've tuned. Key Takeaways - •Our team reviews 70–100 pull requests a week - •This story was reported by Dev.to , covering developments in the dev space. - •AI advancements continue to reshape industries — read the full article on Dev.to for complete coverage. 📖 Continue reading the full article: Read Full Article on Dev.to → https://dev.to/avaneeshyadav/ai-code-review-at-scale-how-we-use-claude-to-review-every-pr-before-humans-see-it-170o