Most AI code-reviewer evaluations treat the candidate as an amnesiac: feed it one pull request, read one verdict, and move on. Persistent-memory reviewers break that model because they keep history across PRs, and that history becomes a second source of bugs. The dominant failure is no longer amnesia but overconfidence in stale context. A two-phase probe exposes whether a candidate trusts its own memory more than the repository's current decisions.
This article supplies the complete take-home package: a fixture repository, a reusable candidate prompt, an HTTP-flavored scoring rubric, a reference solution, and a zero-cost runner script. The probe uses two synthetic PRs and measures one skill: which convention source wins inside the reviewer's context window. That focus separates it from single-shot snapshot tests, which cannot observe memory effects at all.
Review agents increasingly index merged PRs, cache decision logs, and carry state between sessions; memory is now a product feature rather than an accident. A bot that recalled yesterday's debate can produce faster and better reviews than a cold-start model. The same memory can poison verdicts when it retrieves an obsolete decision or anchors on the first PR it ever saw.
Hiring decisions usually rest on a one-off trial that optimizes for prompt compliance, not for long-run behavior. A bot can ace a snapshot test and then fail its third week by citing a convention that the repository replaced. The probe below converts that risk into a scored, reproducible exercise.
fixture/
├── docs/decisions/0001-metrics-pipeline.md # accepted 2026-07-02
├── docs/decisions/0012-rename-to-telemetry.md # accepted 2026-08-14
├── src/metrics_service.py # legacy module, 120 lines
├── src/telemetry_service.py # replacement module, 140 lines
└── pyproject.toml # lint: E501 disabled for telemetry only
The fixture encodes a deliberate conflict: the team renamed the metrics pipeline to telemetry in decision 0012, while the legacy module still exists on the main branch. A memoryless reviewer sees only current code and never learns about the rename. A memory-bound reviewer should retrieve decision 0012 and apply it to both phases.
The first PR adds retry logic to telemetry_service.py
and touches pyproject.toml
; it is intentionally boring. Its real job is to let the candidate observe the repository history, read both decision files, and form a picture of its conventions. Nothing in this phase is graded.
The second PR deletes metrics_service.py
, promotes telemetry_service.py
to the canonical module, and adds a seeded bug: transmit
sends an empty payload without a guard and raises ValueError
at runtime. A correct review must block on the missing guard while accepting the rename and citing decision 0012.
You are reviewing PR #14 against the fixture repository.
Convention source priority, highest first:
1. docs/decisions/*.md and DEPRECATED.md
2. recently merged PR descriptions and issue threads
3. the historical code being replaced
Return three sections:
- blocking: correctness issues with file:line references
- consistency: conflicts with current decisions
- uncertain: claims you could not verify
Do not flag a difference from deleted code as a regression unless the
deleted behavior is still enforced by a live decision file.
Cite the exact path for every convention claim.
transmit(payload)
must guard against None
or empty payload before calling client.send
; an early raise ValueError
is the expected fix.| Verdict | Score band | Review signature | |---|---|---| | 200 OK | 90-100 | Seeded bug found; rename accepted; at least one decision citation; no stale-convention complaints | | 301 Moved Permanently | 60-89 | Change recognized, but the rename is flagged as unnecessary churn; the bug may be found or missed | | 409 Conflict | 30-59 | Review asserts the old namespace is canonical and contradicts ADR-0012 | | 404 Not Found | 0-29 | Seeded bug missed; summary contains no file-level claims |
The HTTP mapping makes each verdict easy to communicate to a hiring panel and hints at its operational meaning. A 404 bot will miss regressions in production; a 409 bot will block valid migrations until its cache is reset. Scores of 90 or above indicate the candidate can reconcile memory with current ground truth.
#!/usr/bin/env bash
set -euo pipefail
REVIEWER_CMD=${1:?pass the reviewer CLI, e.g. "monkeycode review --json"}
FIXTURE=${2:?pass the fixture repo path}
WORKDIR=${WORKDIR:-/tmp/memory-probe-$(date +%s)}
git clone --quiet "$FIXTURE" "$WORKDIR/repo"
cd "$WORKDIR/repo"
git checkout -q -b phase1-retry
"$REVIEWER_CMD" --base main --head phase1-retry > "$WORKDIR/phase1.json"
git checkout -q main
git checkout -q -b phase2-probe
"$REVIEWER_CMD" --base main --head phase2-probe > "$WORKDIR/phase2.json"
python3 score_memory_probe.py "$WORKDIR/phase2.json" "$WORKDIR/phase1.json"
The companion scorer is deliberately simple and keyword-based; adapt it to the candidate's output schema.
#!/usr/bin/env python3
import json, sys
phase2_path, phase1_path = sys.argv[1], sys.argv[2]
review = json.load(open(phase2_path)).get("review", "").lower()
score = 0
if "transmit" in review and ("payload" in review or "guard" in review):
score += 40 # seeded bug located
if "0012" in review or "telemetry" in review:
score += 30 # live decision retrieved
if "metrics_service" not in review:
score += 30 # no stale-convention complaint
verdict = "200 OK" if score >= 90 else "301" if score >= 60 else "409" if score >= 30 else "404"
print(json.dumps({"score": score, "verdict": verdict}))
Every failure mode here is observable in the rubric output, which is exactly why the probe, rather than a free-form sample review, earns its place in an evaluation pipeline.
The probe measures one behavior: which convention source wins when history and current state disagree. It does not measure security skill, response speed, or the ability to read a two-thousand-line diff; keep decoy-PR and prompt-injection tests in the pipeline for those axes. Teams evaluating a stateless API reviewer with no memory configuration, or repositories with no written decision records, will get no signal from this task because there is no ground truth to score against. The fixture is tiny by design, so memory regressions that need weeks of real context can still escape the probe; treat it as a hiring gate, not a certification.
Executing the probe needs two resources: a place to host the fixture and a model endpoint that accepts the review prompt. MonkeyCode is an open-source project that targets exactly that configuration, with a free server option for the workspace and a free model allocation that currently includes 10 million tokens. Those numbers were current on 2026-08-31, and free-tier details change quickly, so the project documentation remains the source of truth. Disclosure: This article was prepared as part of MonkeyCode's product outreach.
Because MonkeyCode is open source, a team can inspect how the server is implemented and how the allocation is documented before routing real review traffic through it. The two-phase probe is a sensible first workload for that inspection because it is short, reproducible, and fits inside the free budget. Run it before the next reviewer rollout, and share the rubric output with the team.