When Your AI Reviewer Remembers Too Much: A Two-Phase Memory Probe

A developer created a two-phase memory probe to evaluate AI code reviewers that retain context across pull requests, exposing a failure mode where bots trust stale memory over current repository decisions. The probe includes a fixture repository, a reusable candidate prompt, a scoring rubric, and a runner script, and it tests whether a reviewer correctly applies a documented rename while catching a seeded bug. The developer warns that memory-based review agents can ace single-shot tests but fail long-term by citing obsolete conventions.

Most AI code-reviewer evaluations treat the candidate as an amnesiac: feed it one pull request, read one verdict, and move on. Persistent-memory reviewers break that model because they keep history across PRs, and that history becomes a second source of bugs. The dominant failure is no longer amnesia but overconfidence in stale context. A two-phase probe exposes whether a candidate trusts its own memory more than the repository's current decisions. This article supplies the complete take-home package: a fixture repository, a reusable candidate prompt, an HTTP-flavored scoring rubric, a reference solution, and a zero-cost runner script. The probe uses two synthetic PRs and measures one skill: which convention source wins inside the reviewer's context window. That focus separates it from single-shot snapshot tests, which cannot observe memory effects at all. Review agents increasingly index merged PRs, cache decision logs, and carry state between sessions; memory is now a product feature rather than an accident. A bot that recalled yesterday's debate can produce faster and better reviews than a cold-start model. The same memory can poison verdicts when it retrieves an obsolete decision or anchors on the first PR it ever saw. Hiring decisions usually rest on a one-off trial that optimizes for prompt compliance, not for long-run behavior. A bot can ace a snapshot test and then fail its third week by citing a convention that the repository replaced. The probe below converts that risk into a scored, reproducible exercise. fixture/ ├── docs/decisions/0001-metrics-pipeline.md accepted 2026-07-02 ├── docs/decisions/0012-rename-to-telemetry.md accepted 2026-08-14 ├── src/metrics service.py legacy module, 120 lines ├── src/telemetry service.py replacement module, 140 lines └── pyproject.toml lint: E501 disabled for telemetry only The fixture encodes a deliberate conflict: the team renamed the metrics pipeline to telemetry in decision 0012, while the legacy module still exists on the main branch. A memoryless reviewer sees only current code and never learns about the rename. A memory-bound reviewer should retrieve decision 0012 and apply it to both phases. The first PR adds retry logic to telemetry service.py and touches pyproject.toml ; it is intentionally boring. Its real job is to let the candidate observe the repository history, read both decision files, and form a picture of its conventions. Nothing in this phase is graded. The second PR deletes metrics service.py , promotes telemetry service.py to the canonical module, and adds a seeded bug: transmit sends an empty payload without a guard and raises ValueError at runtime. A correct review must block on the missing guard while accepting the rename and citing decision 0012. You are reviewing PR 14 against the fixture repository. Convention source priority, highest first: 1. docs/decisions/ .md and DEPRECATED.md 2. recently merged PR descriptions and issue threads 3. the historical code being replaced Return three sections: - blocking: correctness issues with file:line references - consistency: conflicts with current decisions - uncertain: claims you could not verify Do not flag a difference from deleted code as a regression unless the deleted behavior is still enforced by a live decision file. Cite the exact path for every convention claim. transmit payload must guard against None or empty payload before calling client.send ; an early raise ValueError is the expected fix.| Verdict | Score band | Review signature | |---|---|---| | 200 OK | 90-100 | Seeded bug found; rename accepted; at least one decision citation; no stale-convention complaints | | 301 Moved Permanently | 60-89 | Change recognized, but the rename is flagged as unnecessary churn; the bug may be found or missed | | 409 Conflict | 30-59 | Review asserts the old namespace is canonical and contradicts ADR-0012 | | 404 Not Found | 0-29 | Seeded bug missed; summary contains no file-level claims | The HTTP mapping makes each verdict easy to communicate to a hiring panel and hints at its operational meaning. A 404 bot will miss regressions in production; a 409 bot will block valid migrations until its cache is reset. Scores of 90 or above indicate the candidate can reconcile memory with current ground truth. bash /usr/bin/env bash run memory probe.sh — two-phase AI reviewer probe abridged reference harness set -euo pipefail REVIEWER CMD=${1:?pass the reviewer CLI, e.g. "monkeycode review --json"} FIXTURE=${2:?pass the fixture repo path} WORKDIR=${WORKDIR:-/tmp/memory-probe-$ date +%s } git clone --quiet "$FIXTURE" "$WORKDIR/repo" cd "$WORKDIR/repo" Phase 1: boring retry refactor; lets the bot read repository history git checkout -q -b phase1-retry ... apply the retry commit from the task kit ... "$REVIEWER CMD" --base main --head phase1-retry "$WORKDIR/phase1.json" Phase 2: rename plus seeded bug, both hidden in one diff git checkout -q main git checkout -q -b phase2-probe ... apply the probe commit from the task kit ... "$REVIEWER CMD" --base main --head phase2-probe "$WORKDIR/phase2.json" python3 score memory probe.py "$WORKDIR/phase2.json" "$WORKDIR/phase1.json" The companion scorer is deliberately simple and keyword-based; adapt it to the candidate's output schema. python /usr/bin/env python3 import json, sys phase2 path, phase1 path = sys.argv 1 , sys.argv 2 review = json.load open phase2 path .get "review", "" .lower score = 0 if "transmit" in review and "payload" in review or "guard" in review : score += 40 seeded bug located if "0012" in review or "telemetry" in review: score += 30 live decision retrieved if "metrics service" not in review: score += 30 no stale-convention complaint verdict = "200 OK" if score = 90 else "301" if score = 60 else "409" if score = 30 else "404" print json.dumps {"score": score, "verdict": verdict} Every failure mode here is observable in the rubric output, which is exactly why the probe, rather than a free-form sample review, earns its place in an evaluation pipeline. The probe measures one behavior: which convention source wins when history and current state disagree. It does not measure security skill, response speed, or the ability to read a two-thousand-line diff; keep decoy-PR and prompt-injection tests in the pipeline for those axes. Teams evaluating a stateless API reviewer with no memory configuration, or repositories with no written decision records, will get no signal from this task because there is no ground truth to score against. The fixture is tiny by design, so memory regressions that need weeks of real context can still escape the probe; treat it as a hiring gate, not a certification. Executing the probe needs two resources: a place to host the fixture and a model endpoint that accepts the review prompt. MonkeyCode is an open-source project that targets exactly that configuration, with a free server option for the workspace and a free model allocation that currently includes 10 million tokens. Those numbers were current on 2026-08-31, and free-tier details change quickly, so the project documentation remains the source of truth. Disclosure: This article was prepared as part of MonkeyCode's product outreach. Because MonkeyCode is open source, a team can inspect how the server is implemented and how the allocation is documented before routing real review traffic through it. The two-phase probe is a sensible first workload for that inspection because it is short, reproducible, and fits inside the free budget. Run it before the next reviewer rollout, and share the rubric output with the team.