How I built an AI code reviewer that knows when to shut up

A developer built an AI code review tool that enforces conciseness in application code rather than through prompts, using a GitHub webhook, diff fetching, Claude, structured JSON findings, and a severity-based ranker that caps output with a slice() after sorting. Findings are validated against a four-value severity whitelist (BUG, WARN, NIT, PRAISE), stable-sorted, and capped at MAX_COMMENTS, with the suppressed count disclosed in the review body so a quiet reviewer isn't mistaken for one that missed issues. The developer notes the cap can work against PRs with more than eight genuine bugs and that unanchored inline comments fall back to a single top-level comment.

Every AI code reviewer I tried had the same problem: it wouldn't stop talking. Rename this. Add a comment here. Consider extracting that. By the third file you've stopped reading, and a tool you've stopped reading is worse than no tool — it's a tool that will hide a real bug from you inside a wall of suggestions. So I built one with the opposite rule: say nothing unless you found something worth saying. That sounds like a prompt engineering problem. It isn't. The model will happily agree to be concise and then hand you fourteen findings anyway. Every constraint that actually held up in production is a constraint I enforce in application code, after the model has already spoken. Here's what that looks like, including three bugs I only found by pointing the thing at real pull requests and a real credit card. A reviewer you've muted is worse than no reviewer, because now there's a wall of suggestions for a real bug to hide behind. This matters most for solo developers and small teams — the people who don't already have an enterprise code-review bundle sitting on top of their existing tools. If the review output is noisy, they turn it off in week one and never come back. GitHub webhook → diff fetch → Claude → structured findings → ranker → inline comments Every stage after "Claude" exists to decide what not to show you. The model returns JSON findings, each with { path, line, severity, message } , where severity is one of four values: BUG | WARN | NIT | PRAISE . The severity string coming back from the model is validated against that whitelist — an invalid value gets the finding dropped rather than trusted. Then everything is stable-sorted by severity: js const severityRank: Record<ReviewComment "severity" , number = { BUG: 0, WARN: 1, NIT: 2, PRAISE: 3, }; const rankedComments = sanitizedComments .map comment, index = { comment, index } .sort a, b = { const rankDiff = severityRank a.comment.severity - severityRank b.comment.severity ; if rankDiff == 0 return rankDiff; return a.index - b.index; // same-severity ties keep the model's own order } .map entry = entry.comment ; Stable sort matters here: within the same severity, findings keep the order the model produced them in, instead of being silently reshuffled. js const comments = rankedComments.slice 0, MAX COMMENTS ; const suppressedCount = sanitizedComments.length - comments.length; MAX COMMENTS is a constant, applied by slice after the sort — not a line in the prompt asking the model to "please be concise." A prompt is a request the model can ignore on a bad day; a slice cannot. What gets cut isn't dropped silently. suppressedCount flows into the review body: body += 🔇 ${result.suppressedCount} lower-priority remark${result.suppressedCount == 1 ? "s" : ""} suppressed to keep this review focused.\n\n ; Why disclose the count instead of just trimming quietly? Because "quiet reviewer" and "reviewer that missed it" look identical from the outside unless something tells you which one you're looking at. Fair pushback on this design, and I don't have a clean answer: on a PR with more than 8 genuine bugs, the cap works against you. Sorting bugs to the front means you at least see the worst of it first, but "the cap doesn't hide real bugs" is not a guarantee I'm willing to write — only that severity ordering makes it less likely. There's a fourth severity that isn't a problem at all — a slot reserved for "this was a good change." A review that's only ever negative gets the same treatment as a chatty one: people stop opening it. GitHub's inline PR comments only land if the position matches the actual diff hunk. If a finding's line doesn't anchor, the naive move is to drop it. Instead, the handler falls back to a single top-level comment that lists everything, formatted findings included: try { await createReview / ...inline comments... / ; } catch reviewError { console.error "Inline review failed, falling back to issue comment:", reviewError ; await postPRComment octokit, owner, repo, prNumber, fallbackCommentBody reviewResult ; } A formatting mismatch shouldn't be able to delete a finding. GitHub wants a 2XX response within roughly 10 seconds of a webhook delivery. Generating a review — fetch the diff, call the model, post the comments — routinely takes longer than that. Doing all of it synchronously meant GitHub logged the delivery as failed while my function kept running and posted the comments anyway. The delivery log said "failed." The PR said otherwise. Both were technically correct, which made it maddening to debug. The fix: verify the signature, return 200 immediately, and do the actual work in the background. case "pull request": waitUntil handlePullRequestEvent payload .catch error = { console.error "PR review background processing failed:", { deliveryId }, error ; } ; break; Async fixed the timeout problem and introduced a worse one: if the background work dies partway through, the review row just sits at "pending" forever. Not visible on the dashboard as an error. Not counted against quota. GitHub already has its 200. Nothing anywhere tells you a review didn't happen — that's the definition of a silent failure. The fix was to stop depending on platform defaults and make the ceiling explicit: // Give the background work PR file fetch + Claude review + GitHub post // enough time. Leaving this unset silently inherits Vercel's default, // and on timeout the review sits at "pending" with no way to detect it. export const maxDuration = 120; Going async doesn't remove failure, it changes what failure looks like. Anything that runs outside the request/response cycle needs its own explicit way of surfacing "this didn't finish" — a timeout, a status you can query, something. Silent pending isn't good enough. This one wasn't a code review bug, it was a billing bug, and I only found it because I ran my own Stripe checkout on the live account. Cancelled a trial from the customer portal. Stripe's own dashboard confirmed "cancels date ." My app's dashboard kept saying "Renews." The webhook handler was trusting cancel at period end as the single source of truth for "is this cancelling": cancelAtPeriodEnd: sub.cancel at period end, // before the fix Pulling the actual webhook payload in the Stripe dashboard showed the real shape of the event: cancel at period end stayed false through the whole cancellation, while cancel at flipped from null to a real timestamp. Current Stripe subscription behavior resolves an end-of-period cancellation directly to a cancel at timestamp rather than only flipping the boolean. The fix reads both: // cancel at period end alone isn't reliable here — cancellation can resolve // straight to a cancel at timestamp instead. Check both, keep the boolean // path for backward compatibility. cancelAtPeriodEnd: sub.cancel at period end || sub.cancel at = null, Two things I took from this: don't trust a single boolean field to represent a state transition without checking the real payload first, and billing paths get tested with a real card, not a happy-path assumption about what the API returns. Six lines to fix, but shipped as-is it would have told every cancelling customer a lie about their own subscription. DevReview reviews GitHub pull requests and ranks findings BUG → WARN → NIT → PRAISE, hard-capped at 8 comments per review with the suppressed count disclosed rather than hidden. Free tier is $0 forever — 15 reviews/month on one repo, no card involved. Pro is $9/user/month with a 14-day trial card required at checkout, first charge on day 15 . I'd genuinely like to know where the 8-comment cap is the wrong call — tell me if you try it.