Your AI Code Reviewer Needs a Test Suite Too

A developer proposes treating LLM-based code reviewers as production systems that need their own test suites, using labeled PR fixtures and pytest to assert on reviewer behavior rather than exact wording. The approach pairs known-bad diffs with expected.yaml manifests specifying must_flag and must_not_flag findings, and pushes for structured JSON output via Pydantic schemas to keep assertions from becoming brittle. The author notes that prompt tweaks, model version bumps, and system-prompt edits can silently degrade a reviewer's security coverage without anyone noticing.

You plugged an LLM into your PR pipeline. It leaves comments about SQL injection, N+1 queries, missing null checks. Everyone's thrilled for about two sprints, and then someone notices it stopped catching the exact same bug pattern it used to flag. A prompt got tweaked. A model version got bumped upstream. Someone added "be more concise" to the system prompt and it started skipping the security section entirely. Nobody noticed because nobody was checking. The AI reviewer is running in production, making judgment calls on every PR in your org, and it has zero test coverage. That's the part that should bother you more than it probably does. We test the code the AI reviews. We don't test the reviewer itself, because it feels fuzzy — "it's an LLM, how do you even assert against prose?" But that's a cop-out. You don't need to assert on exact wording. You need to assert on behavior : given a diff with a known bug, does the reviewer flag it, in the right file, with the right severity, without three false positives burying the real issue? That's a testable claim. It just requires building fixtures the same way you'd build fixtures for any other system with non-deterministic-ish output — like testing a search ranking algorithm or a fraud-detection model. The core idea: collect real PRs or synthetic ones with known, labeled defects, and treat them like golden files. Run each one through your reviewer, then assert the output contains — or doesn't contain — specific findings. Here's a minimal structure using Python and pytest, since most review pipelines are just a script wrapping an LLM call: python import os from openai import OpenAI client = OpenAI api key=os.environ "OPENAI API KEY" SYSTEM PROMPT = open "prompts/review system.md" .read def review diff diff text: str - str: response = client.chat.completions.create model="gpt-4.1", temperature=0, messages= {"role": "system", "content": SYSTEM PROMPT}, {"role": "user", "content": f"Review this diff:\n\n{diff text}"}, , return response.choices 0 .message.content Note temperature=0 . You're not eliminating nondeterminism, but you're minimizing it — this matters a lot when you're about to assert against the output. Each fixture is a known-bad diff plus a manifest describing what should get flagged: fixtures/ sql injection raw query/ diff.patch expected.yaml missing await async call/ diff.patch expected.yaml hardcoded secret in config/ diff.patch expected.yaml yaml must flag: The must not flag block matters as much as must flag . A reviewer that comments on everything technically "catches" every bug, but it's useless noise. You want to pin down both precision and recall. Since the output is prose, not structured data, you have two options: force structured output JSON mode, function calling or parse loosely with keyword/semantic matching. I'd push hard for structured output — it makes the whole test suite dramatically less brittle. from pydantic import BaseModel class Finding BaseModel : category: str file: str line start: int line end: int message: str severity: str class ReviewResult BaseModel : findings: list Finding import yaml import pytest from pathlib import Path from reviewer.client import review diff from reviewer.schema import ReviewResult FIXTURE DIR = Path "fixtures" def load fixtures : for folder in FIXTURE DIR.iterdir : diff = folder / "diff.patch" .read text expected = yaml.safe load folder / "expected.yaml" .read text yield pytest.param diff, expected, id=folder.name @pytest.mark.parametrize "diff,expected", load fixtures def test reviewer catches known bug diff, expected : raw = review diff diff result = ReviewResult.model validate json raw for must in expected.get "must flag", : matches = f for f in result.findings if f.category == must "category" and f.file == must "file" and any kw in f.message.lower for kw in must "keywords" assert matches, f"Reviewer missed expected finding: {must}" for forbidden in expected.get "must not flag", : matches = f for f in result.findings if any kw in f.message.lower for kw in forbidden "keywords" assert not matches, f"Reviewer raised noise it shouldn't have: {forbidden}" max comments = expected.get "max total comments" if max comments is not None: assert len result.findings <= max comments This is a regression suite in the truest sense: every time someone edits the system prompt, swaps the model, or adjusts temperature, this runs and tells you exactly what broke. "Prompt change reduced recall on SQL injection cases from 100% to 60%" is a real, actionable CI failure — not a vibe. The expensive part is the LLM calls, so don't run this on every commit to every branch. Run it: prompts/ , reviewer/ , or model config name: reviewer-regression on: pull request: paths: - "prompts/ " - "reviewer/ " This isn't free, and it's not perfectly deterministic even with temperature=0 — different model versions can still shift slightly. A few honest trade-offs: None of these are reasons to skip this. They're reasons to scope it like any other test suite — start with the five bug patterns that have bitten you hardest in production, not fifty hypothetical ones. If your AI reviewer has opinions about your code quality, it deserves the same scrutiny you'd apply to any other piece of logic sitting between a developer and a merge button. A golden-diff suite is cheap to start — a handful of real bugs pulled from your git history and a YAML file describing what "catching it" looks like. What's the bug pattern your AI reviewer has already let through that you haven't turned into a test case yet? That's usually fixture 1.