# Your AI Code Reviewer Needs a Test Suite Too

> Source: <https://dev.to/renato_silva_71eef0fc385f/your-ai-code-reviewer-needs-a-test-suite-too-4g41>
> Published: 2026-09-24 13:58:17+00:00

You plugged an LLM into your PR pipeline. It leaves comments about SQL injection, N+1 queries, missing null checks. Everyone's thrilled for about two sprints, and then someone notices it stopped catching the exact same bug pattern it used to flag. A prompt got tweaked. A model version got bumped upstream. Someone added "be more concise" to the system prompt and it started skipping the security section entirely.

Nobody noticed because nobody was checking. The AI reviewer is running in production, making judgment calls on every PR in your org, and it has zero test coverage. That's the part that should bother you more than it probably does.

We test the code the AI reviews. We don't test the reviewer itself, because it feels fuzzy — "it's an LLM, how do you even assert against prose?" But that's a cop-out. You don't need to assert on exact wording. You need to assert on *behavior*: given a diff with a known bug, does the reviewer flag it, in the right file, with the right severity, without three false positives burying the real issue?

That's a testable claim. It just requires building fixtures the same way you'd build fixtures for any other system with non-deterministic-ish output — like testing a search ranking algorithm or a fraud-detection model.

The core idea: collect real PRs (or synthetic ones) with known, labeled defects, and treat them like golden files. Run each one through your reviewer, then assert the output contains — or doesn't contain — specific findings.

Here's a minimal structure using Python and pytest, since most review pipelines are just a script wrapping an LLM call:

python

import os

from openai import OpenAI

client = OpenAI(api_key=os.environ["OPENAI_API_KEY"])

SYSTEM_PROMPT = open("prompts/review_system.md").read()

def review_diff(diff_text: str) -> str:

    response = client.chat.completions.create(

        model="gpt-4.1",

        temperature=0,

        messages=[

            {"role": "system", "content": SYSTEM_PROMPT},

            {"role": "user", "content": f"Review this diff:\n\n{diff_text}"},

        ],

    )

    return response.choices[0].message.content

Note `temperature=0`. You're not eliminating nondeterminism, but you're minimizing it — this matters a lot when you're about to assert against the output.

Each fixture is a known-bad diff plus a manifest describing what *should* get flagged:

fixtures/

  sql_injection_raw_query/

    diff.patch

    expected.yaml

  missing_await_async_call/

    diff.patch

    expected.yaml

  hardcoded_secret_in_config/

    diff.patch

    expected.yaml

yaml

must_flag:

The `must_not_flag` block matters as much as `must_flag`. A reviewer that comments on everything technically "catches" every bug, but it's useless noise. You want to pin down both precision and recall.

Since the output is prose, not structured data, you have two options: force structured output (JSON mode, function calling) or parse loosely with keyword/semantic matching. I'd push hard for structured output — it makes the whole test suite dramatically less brittle.

from pydantic import BaseModel

class Finding(BaseModel):

    category: str

    file: str

    line_start: int

    line_end: int

    message: str

    severity: str

class ReviewResult(BaseModel):

    findings: list[Finding]

import yaml

import pytest

from pathlib import Path

from reviewer.client import review_diff

from reviewer.schema import ReviewResult

FIXTURE_DIR = Path("fixtures")

def load_fixtures():

    for folder in FIXTURE_DIR.iterdir():

        diff = (folder / "diff.patch").read_text()

        expected = yaml.safe_load((folder / "expected.yaml").read_text())

        yield pytest.param(diff, expected, id=folder.name)

@pytest.mark.parametrize("diff,expected", load_fixtures())

def test_reviewer_catches_known_bug(diff, expected):

    raw = review_diff(diff)

    result = ReviewResult.model_validate_json(raw)

```
for must in expected.get("must_flag", []):
    matches = [
        f for f in result.findings
        if f.category == must["category"]
        and f.file == must["file"]
        and any(kw in f.message.lower() for kw in must["keywords"])
    ]
    assert matches, f"Reviewer missed expected finding: {must}"

for forbidden in expected.get("must_not_flag", []):
    matches = [
        f for f in result.findings
        if any(kw in f.message.lower() for kw in forbidden["keywords"])
    ]
    assert not matches, f"Reviewer raised noise it shouldn't have: {forbidden}"

max_comments = expected.get("max_total_comments")
if max_comments is not None:
    assert len(result.findings) <= max_comments
```

This is a regression suite in the truest sense: every time someone edits the system prompt, swaps the model, or adjusts temperature, this runs and tells you exactly what broke. "Prompt change reduced recall on SQL injection cases from 100% to 60%" is a real, actionable CI failure — not a vibe.

The expensive part is the LLM calls, so don't run this on every commit to every branch. Run it:

`prompts/`, `reviewer/`, or model config
name: reviewer-regression

on:

  pull_request:

    paths:

      - "prompts/**" - "reviewer/**"

This isn't free, and it's not perfectly deterministic even with `temperature=0` — different model versions can still shift slightly. A few honest trade-offs:

None of these are reasons to skip this. They're reasons to scope it like any other test suite — start with the five bug patterns that have bitten you hardest in production, not fifty hypothetical ones.

If your AI reviewer has opinions about your code quality, it deserves the same scrutiny you'd apply to any other piece of logic sitting between a developer and a merge button. A golden-diff suite is cheap to start — a handful of real bugs pulled from your git history and a YAML file describing what "catching it" looks like.

What's the bug pattern your AI reviewer has already let through that you haven't turned into a test case yet? That's usually fixture #1.