{"slug": "how-i-built-an-ai-code-reviewer-that-knows-when-to-shut-up", "title": "How I built an AI code reviewer that knows when to shut up", "summary": "A developer built an AI code review tool that enforces conciseness in application code rather than through prompts, using a GitHub webhook, diff fetching, Claude, structured JSON findings, and a severity-based ranker that caps output with a slice() after sorting. Findings are validated against a four-value severity whitelist (BUG, WARN, NIT, PRAISE), stable-sorted, and capped at MAX_COMMENTS, with the suppressed count disclosed in the review body so a quiet reviewer isn't mistaken for one that missed issues. The developer notes the cap can work against PRs with more than eight genuine bugs and that unanchored inline comments fall back to a single top-level comment.", "body_md": "Every AI code reviewer I tried had the same problem: it wouldn't stop talking.\n\nRename this. Add a comment here. Consider extracting that. By the third file\n\nyou've stopped reading, and a tool you've stopped reading is worse than no\n\ntool — it's a tool that will hide a real bug from you inside a wall of\n\nsuggestions.\n\nSo I built one with the opposite rule: say nothing unless you found something\n\nworth saying. That sounds like a prompt engineering problem. It isn't. The\n\nmodel will happily agree to be concise and then hand you fourteen findings\n\nanyway. Every constraint that actually held up in production is a constraint\n\nI enforce in application code, after the model has already spoken.\n\nHere's what that looks like, including three bugs I only found by pointing\n\nthe thing at real pull requests and a real credit card.\n\nA reviewer you've muted is worse than no reviewer, because now there's a wall\n\nof suggestions for a real bug to hide behind. This matters most for solo\n\ndevelopers and small teams — the people who don't already have an enterprise\n\ncode-review bundle sitting on top of their existing tools. If the review\n\noutput is noisy, they turn it off in week one and never come back.\n\n```\nGitHub webhook → diff fetch → Claude → structured findings → ranker → inline comments\n```\n\nEvery stage after \"Claude\" exists to decide what *not* to show you.\n\nThe model returns JSON findings, each with `{ path, line, severity, message }`,\n\nwhere severity is one of four values: `BUG | WARN | NIT | PRAISE`. The\n\nseverity string coming back from the model is validated against that\n\nwhitelist — an invalid value gets the finding dropped rather than trusted.\n\nThen everything is stable-sorted by severity:\n\n``` js\nconst severityRank: Record<ReviewComment[\"severity\"], number> = {\n  BUG: 0,\n  WARN: 1,\n  NIT: 2,\n  PRAISE: 3,\n};\nconst rankedComments = sanitizedComments\n  .map((comment, index) => ({ comment, index }))\n  .sort((a, b) => {\n    const rankDiff = severityRank[a.comment.severity] - severityRank[b.comment.severity];\n    if (rankDiff !== 0) return rankDiff;\n    return a.index - b.index; // same-severity ties keep the model's own order\n  })\n  .map((entry) => entry.comment);\n```\n\nStable sort matters here: within the same severity, findings keep the order\n\nthe model produced them in, instead of being silently reshuffled.\n\n``` js\nconst comments = rankedComments.slice(0, MAX_COMMENTS);\nconst suppressedCount = sanitizedComments.length - comments.length;\n```\n\n`MAX_COMMENTS` is a constant, applied by `slice` after the sort — not a line\n\nin the prompt asking the model to \"please be concise.\" A prompt is a request\n\nthe model can ignore on a bad day; a `slice()` cannot.\n\nWhat gets cut isn't dropped silently. `suppressedCount` flows into the review\n\nbody:\n\n```\nbody += `🔇 ${result.suppressedCount} lower-priority remark${result.suppressedCount !== 1 ? \"s\" : \"\"} suppressed to keep this review focused.\\n\\n`;\n```\n\nWhy disclose the count instead of just trimming quietly? Because \"quiet\n\nreviewer\" and \"reviewer that missed it\" look identical from the outside\n\nunless something tells you which one you're looking at.\n\nFair pushback on this design, and I don't have a clean answer: on a PR with\n\nmore than 8 genuine bugs, the cap works against you. Sorting bugs to the\n\nfront means you at least see the worst of it first, but \"the cap doesn't\n\nhide real bugs\" is not a guarantee I'm willing to write — only that severity\n\nordering makes it less likely.\n\nThere's a fourth severity that isn't a problem at all — a slot reserved for\n\n\"this was a good change.\" A review that's only ever negative gets the same\n\ntreatment as a chatty one: people stop opening it.\n\nGitHub's inline PR comments only land if the position matches the actual\n\ndiff hunk. If a finding's line doesn't anchor, the naive move is to drop it.\n\nInstead, the handler falls back to a single top-level comment that lists\n\neverything, formatted findings included:\n\n```\ntry {\n  await createReview(/* ...inline comments... */);\n} catch (reviewError) {\n  console.error(\"Inline review failed, falling back to issue comment:\", reviewError);\n  await postPRComment(octokit, owner, repo, prNumber, fallbackCommentBody(reviewResult));\n}\n```\n\nA formatting mismatch shouldn't be able to delete a finding.\n\nGitHub wants a 2XX response within roughly 10 seconds of a webhook delivery.\n\nGenerating a review — fetch the diff, call the model, post the comments —\n\nroutinely takes longer than that. Doing all of it synchronously meant GitHub\n\nlogged the delivery as failed while my function kept running and posted the\n\ncomments anyway. The delivery log said \"failed.\" The PR said otherwise. Both\n\nwere technically correct, which made it maddening to debug.\n\nThe fix: verify the signature, return 200 immediately, and do the actual\n\nwork in the background.\n\n```\ncase \"pull_request\":\n  waitUntil(\n    handlePullRequestEvent(payload).catch((error) => {\n      console.error(\"PR review background processing failed:\", { deliveryId }, error);\n    })\n  );\n  break;\n```\n\nAsync fixed the timeout problem and introduced a worse one: if the\n\nbackground work dies partway through, the review row just sits at\n\n`\"pending\"` forever. Not visible on the dashboard as an error. Not counted\n\nagainst quota. GitHub already has its 200. Nothing anywhere tells you a\n\nreview didn't happen — that's the definition of a silent failure.\n\nThe fix was to stop depending on platform defaults and make the ceiling\n\nexplicit:\n\n```\n// Give the background work (PR file fetch + Claude review + GitHub post)\n// enough time. Leaving this unset silently inherits Vercel's default,\n// and on timeout the review sits at \"pending\" with no way to detect it.\nexport const maxDuration = 120;\n```\n\nGoing async doesn't remove failure, it changes what failure looks like.\n\nAnything that runs outside the request/response cycle needs its own\n\nexplicit way of surfacing \"this didn't finish\" — a timeout, a status you\n\ncan query, something. Silent pending isn't good enough.\n\nThis one wasn't a code review bug, it was a billing bug, and I only found it\n\nbecause I ran my own Stripe checkout on the live account. Cancelled a trial\n\nfrom the customer portal. Stripe's own dashboard confirmed \"cancels\n\n[date].\" My app's dashboard kept saying \"Renews.\"\n\nThe webhook handler was trusting `cancel_at_period_end` as the single source\n\nof truth for \"is this cancelling\":\n\n```\ncancelAtPeriodEnd: sub.cancel_at_period_end,  // before the fix\n```\n\nPulling the actual webhook payload in the Stripe dashboard showed the real\n\nshape of the event: `cancel_at_period_end` stayed `false` through the whole\n\ncancellation, while `cancel_at` flipped from `null` to a real timestamp.\n\nCurrent Stripe subscription behavior resolves an end-of-period cancellation\n\ndirectly to a `cancel_at` timestamp rather than only flipping the boolean.\n\nThe fix reads both:\n\n```\n// cancel_at_period_end alone isn't reliable here — cancellation can resolve\n// straight to a cancel_at timestamp instead. Check both, keep the boolean\n// path for backward compatibility.\ncancelAtPeriodEnd: sub.cancel_at_period_end || sub.cancel_at != null,\n```\n\nTwo things I took from this: don't trust a single boolean field to represent\n\na state transition without checking the real payload first, and billing\n\npaths get tested with a real card, not a happy-path assumption about what\n\nthe API returns. Six lines to fix, but shipped as-is it would have told\n\nevery cancelling customer a lie about their own subscription.\n\nDevReview reviews GitHub pull requests and ranks findings BUG → WARN → NIT →\n\nPRAISE, hard-capped at 8 comments per review with the suppressed count\n\ndisclosed rather than hidden. Free tier is $0 forever — 15 reviews/month on\n\none repo, no card involved. Pro is $9/user/month with a 14-day trial (card\n\nrequired at checkout, first charge on day 15).\n\nI'd genuinely like to know where the 8-comment cap is the wrong call —\n\ntell me if you try it.", "url": "https://wpnews.pro/news/how-i-built-an-ai-code-reviewer-that-knows-when-to-shut-up", "canonical_source": "https://dev.to/phi_blankslate/how-i-built-an-ai-code-reviewer-that-knows-when-to-shut-up-3b0c", "published_at": "2026-09-18 08:43:34+00:00", "updated_at": "2026-09-18 08:52:55.214333+00:00", "lang": "en", "topics": ["ai-tools", "developer-tools", "ai-agents", "large-language-models"], "entities": ["GitHub", "Claude"], "alternates": {"html": "https://wpnews.pro/news/how-i-built-an-ai-code-reviewer-that-knows-when-to-shut-up", "markdown": "https://wpnews.pro/news/how-i-built-an-ai-code-reviewer-that-knows-when-to-shut-up.md", "text": "https://wpnews.pro/news/how-i-built-an-ai-code-reviewer-that-knows-when-to-shut-up.txt", "jsonld": "https://wpnews.pro/news/how-i-built-an-ai-code-reviewer-that-knows-when-to-shut-up.jsonld"}}