You plugged an LLM into your PR pipeline. It leaves comments about SQL injection, N+1 queries, missing null checks. Everyone's thrilled for about two sprints, and then someone notices it stopped catching the exact same bug pattern it used to flag. A prompt got tweaked. A model version got bumped upstream. Someone added "be more concise" to the system prompt and it started skipping the security section entirely.
Nobody noticed because nobody was checking. The AI reviewer is running in production, making judgment calls on every PR in your org, and it has zero test coverage. That's the part that should bother you more than it probably does.
We test the code the AI reviews. We don't test the reviewer itself, because it feels fuzzy — "it's an LLM, how do you even assert against prose?" But that's a cop-out. You don't need to assert on exact wording. You need to assert on behavior: given a diff with a known bug, does the reviewer flag it, in the right file, with the right severity, without three false positives burying the real issue?
That's a testable claim. It just requires building fixtures the same way you'd build fixtures for any other system with non-deterministic-ish output — like testing a search ranking algorithm or a fraud-detection model.
The core idea: collect real PRs (or synthetic ones) with known, labeled defects, and treat them like golden files. Run each one through your reviewer, then assert the output contains — or doesn't contain — specific findings.
Here's a minimal structure using Python and pytest, since most review pipelines are just a script wrapping an LLM call:
python
import os
from openai import OpenAI
client = OpenAI(api_key=os.environ["OPENAI_API_KEY"])
SYSTEM_PROMPT = open("prompts/review_system.md").read()
def review_diff(diff_text: str) -> str:
response = client.chat.completions.create(
model="gpt-4.1",
temperature=0,
messages=[
{"role": "system", "content": SYSTEM_PROMPT},
{"role": "user", "content": f"Review this diff:\n\n{diff_text}"},
],
)
return response.choices[0].message.content
Note temperature=0. You're not eliminating nondeterminism, but you're minimizing it — this matters a lot when you're about to assert against the output.
Each fixture is a known-bad diff plus a manifest describing what should get flagged:
fixtures/
sql_injection_raw_query/
diff.patch
expected.yaml
missing_await_async_call/
diff.patch
expected.yaml
hardcoded_secret_in_config/
diff.patch
expected.yaml
yaml
must_flag:
The must_not_flag block matters as much as must_flag. A reviewer that comments on everything technically "catches" every bug, but it's useless noise. You want to pin down both precision and recall.
Since the output is prose, not structured data, you have two options: force structured output (JSON mode, function calling) or parse loosely with keyword/semantic matching. I'd push hard for structured output — it makes the whole test suite dramatically less brittle.
from pydantic import BaseModel
class Finding(BaseModel):
category: str
file: str
line_start: int
line_end: int
message: str
severity: str
class ReviewResult(BaseModel):
findings: list[Finding]
import yaml
import pytest
from pathlib import Path
from reviewer.client import review_diff
from reviewer.schema import ReviewResult
FIXTURE_DIR = Path("fixtures")
def load_fixtures():
for folder in FIXTURE_DIR.iterdir():
diff = (folder / "diff.patch").read_text()
expected = yaml.safe_load((folder / "expected.yaml").read_text())
yield pytest.param(diff, expected, id=folder.name)
@pytest.mark.parametrize("diff,expected", load_fixtures())
def test_reviewer_catches_known_bug(diff, expected):
raw = review_diff(diff)
result = ReviewResult.model_validate_json(raw)
for must in expected.get("must_flag", []):
matches = [
f for f in result.findings
if f.category == must["category"]
and f.file == must["file"]
and any(kw in f.message.lower() for kw in must["keywords"])
]
assert matches, f"Reviewer missed expected finding: {must}"
for forbidden in expected.get("must_not_flag", []):
matches = [
f for f in result.findings
if any(kw in f.message.lower() for kw in forbidden["keywords"])
]
assert not matches, f"Reviewer raised noise it shouldn't have: {forbidden}"
max_comments = expected.get("max_total_comments")
if max_comments is not None:
assert len(result.findings) <= max_comments
This is a regression suite in the truest sense: every time someone edits the system prompt, swaps the model, or adjusts temperature, this runs and tells you exactly what broke. "Prompt change reduced recall on SQL injection cases from 100% to 60%" is a real, actionable CI failure — not a vibe.
The expensive part is the LLM calls, so don't run this on every commit to every branch. Run it:
prompts/, reviewer/, or model config
name: reviewer-regression
on:
pull_request:
paths:
- "prompts/**" - "reviewer/**"
This isn't free, and it's not perfectly deterministic even with temperature=0 — different model versions can still shift slightly. A few honest trade-offs:
None of these are reasons to skip this. They're reasons to scope it like any other test suite — start with the five bug patterns that have bitten you hardest in production, not fifty hypothetical ones.
If your AI reviewer has opinions about your code quality, it deserves the same scrutiny you'd apply to any other piece of logic sitting between a developer and a merge button. A golden-diff suite is cheap to start — a handful of real bugs pulled from your git history and a YAML file describing what "catching it" looks like.
What's the bug pattern your AI reviewer has already let through that you haven't turned into a test case yet? That's usually fixture #1.