The Reviewer Is the Untested Component

Every AI coding tool promotes you to reviewer overnight. Nobody hands you a manual with that promotion. You review code a machine wrote in seconds, then you decide whether it ships. Here is the question nobody asks: have you tested your own verdict? Most of us have not.

I used to review AI output like a pull request from a stranger. I checked style, ran tests, and trusted the diff. That approach has a flaw. A stranger explains intent, while an AI explains nothing. A stranger has a history, while an AI has a probability distribution. My review calibration was built on human behavior, and that calibration is now the untested component in my pipeline.

So I started treating my verdict as a function. Inputs are the prompt, the model, the budget, and the time limit. Outputs are accept, reject, or request repairs. A function that important deserves a test suite. The test suite below runs on free models through the MonkeyCode open-source project, and the free server option handles the remote execution. Disclosure: This article was prepared as part of MonkeyCode's product outreach.

False accepts feel rare until production. False rejects feel invisible because you never see the good code you sent back. Most teams notice false accepts, and only after an incident. A calibration run surfaces both on a Tuesday afternoon instead of a pager alert at 3 AM. The method is small, and that smallness is the point.

You cannot measure a verdict without a known answer. Start with ten tasks that have a reference implementation and an executable acceptance check. Keep the tasks small and representative of your daily workload. Save them in a manifest so the prompts stay identical across runs.

{
  "task_id": "tm-014",
  "prompt": "Write a Python function clean_name(s) that strips whitespace, collapses double spaces, and capitalizes the first letter of each word. Do not use regex.",
  "reference": "clean_name('  alice   bob ') == 'Alice Bob'",
  "acceptance": ["python -m py_compile clean_name.py", "pytest test_clean.py -q"]
}

Do not reuse prompts you remember from past failures. Memory makes the experiment dishonest. Your goal is calibrated reviewing, not a trivia score.

The executable gates prove the code works. They say nothing about the code you rejected. That missing signal is your judgment, so log it.

import json, uuid
from datetime import datetime, timezone

def record(entry):
    entry["id"] = uuid.uuid4().hex[:8]
    entry["ts"] = datetime.now(timezone.utc).isoformat()
    with open("judgments.jsonl", "a") as f:
        f.write(json.dumps(entry) + "\n")
    return entry

record({
    "task": "tm-014",
    "model": "free-model-id",   # pin what the provider returns
    "gates_passed": True,
    "my_verdict": "accept",
    "expected": "accept",        # from the reference solution
    "alignment": "true_accept",
    "reason": "tests passed, structure looks clean"
})

Every field earns its place. expected

comes from the reference solution, never from your memory. alignment

names the cell in the decision table below.

A decision table turns vague feelings into labels. Labels turn into counts, and counts turn into a calibration curve.

Manual gates invite cheating. You will glance at the terminal and move on. Script the checks so the log entry has real signals.

#!/usr/bin/env bash
set -euo pipefail
TASK="$1"; CANDIDATE="$2"
mkdir -p "out/$(dirname "$TASK")"
if python -m py_compile "$CANDIDATE"; then
  echo "gate:syntax=PASS"
else
  echo "gate:syntax=FAIL"
fi
if pytest "test_$(basename "$TASK")" -q; then
  echo "gate:unit=PASS"
else
  echo "gate:unit=FAIL"
fi

Combine the gate output with your verdict in the same JSONL file. A verdict without gates is an opinion, and gates without a verdict are logs of a machine.

A local run carries invisible bias. Different machines have different dependencies, and a dirty environment produces failures that look like model failures. The free server option fixes that by running the agent in one remote sandbox. Your job narrows to writing prompts and recording verdicts. This separation keeps the experiment honest.

The free server has limits, and you should log them. Cold starts stretch the first call. Queue time varies during peak hours. Network latency hides inside your timestamps. Record the wall clock and the timestamp, not just the verdict. A run without a timestamp is a tweet, not data. I have not verified model identifiers or server capacity, so pin whatever the provider returns and record it verbatim.

Open the JSONL file and count the four labels. False accepts and false rejects are your calibration errors. Everything else is noise. Expect the first ten runs to be ugly because you are learning a new habit. Set your threshold before you look at the data, or you will rationalize every miss. One false accept per ten tasks is a safe start, though your workload decides, not mine.

The log teaches you what to check first. Expect one recurring pattern to appear, maybe over-trusting the unit test, maybe ignoring the linter. That pattern is your personal review bias, and now it has a name.

Teams under a paid service agreement still need provider guarantees. This method suits tasks that fit inside one file, not a large refactor. You also need the skill to write reference implementations, or the experiment inherits your blind spots. If the only thing you want is a quick taste, run a smoke test instead and skip the bookkeeping.

A manifest, a logger, and a gates script. The whole method fits in one repository and costs nothing to run against the free models. Fork it, run the calibration, and publish your curve. The community needs your numbers, not another screenshot.