{"slug": "when-your-ai-reviewer-remembers-too-much-a-two-phase-memory-probe", "title": "When Your AI Reviewer Remembers Too Much: A Two-Phase Memory Probe", "summary": "A developer created a two-phase memory probe to evaluate AI code reviewers that retain context across pull requests, exposing a failure mode where bots trust stale memory over current repository decisions. The probe includes a fixture repository, a reusable candidate prompt, a scoring rubric, and a runner script, and it tests whether a reviewer correctly applies a documented rename while catching a seeded bug. The developer warns that memory-based review agents can ace single-shot tests but fail long-term by citing obsolete conventions.", "body_md": "Most AI code-reviewer evaluations treat the candidate as an amnesiac: feed it one pull request, read one verdict, and move on. Persistent-memory reviewers break that model because they keep history across PRs, and that history becomes a second source of bugs. The dominant failure is no longer amnesia but overconfidence in stale context. A two-phase probe exposes whether a candidate trusts its own memory more than the repository's current decisions.\n\nThis article supplies the complete take-home package: a fixture repository, a reusable candidate prompt, an HTTP-flavored scoring rubric, a reference solution, and a zero-cost runner script. The probe uses two synthetic PRs and measures one skill: which convention source wins inside the reviewer's context window. That focus separates it from single-shot snapshot tests, which cannot observe memory effects at all.\n\nReview agents increasingly index merged PRs, cache decision logs, and carry state between sessions; memory is now a product feature rather than an accident. A bot that recalled yesterday's debate can produce faster and better reviews than a cold-start model. The same memory can poison verdicts when it retrieves an obsolete decision or anchors on the first PR it ever saw.\n\nHiring decisions usually rest on a one-off trial that optimizes for prompt compliance, not for long-run behavior. A bot can ace a snapshot test and then fail its third week by citing a convention that the repository replaced. The probe below converts that risk into a scored, reproducible exercise.\n\n```\nfixture/\n├── docs/decisions/0001-metrics-pipeline.md    # accepted 2026-07-02\n├── docs/decisions/0012-rename-to-telemetry.md # accepted 2026-08-14\n├── src/metrics_service.py                     # legacy module, 120 lines\n├── src/telemetry_service.py                   # replacement module, 140 lines\n└── pyproject.toml                             # lint: E501 disabled for telemetry only\n```\n\nThe fixture encodes a deliberate conflict: the team renamed the metrics pipeline to telemetry in decision 0012, while the legacy module still exists on the main branch. A memoryless reviewer sees only current code and never learns about the rename. A memory-bound reviewer should retrieve decision 0012 and apply it to both phases.\n\nThe first PR adds retry logic to `telemetry_service.py`\n\nand touches `pyproject.toml`\n\n; it is intentionally boring. Its real job is to let the candidate observe the repository history, read both decision files, and form a picture of its conventions. Nothing in this phase is graded.\n\nThe second PR deletes `metrics_service.py`\n\n, promotes `telemetry_service.py`\n\nto the canonical module, and adds a seeded bug: `transmit`\n\nsends an empty payload without a guard and raises `ValueError`\n\nat runtime. A correct review must block on the missing guard while accepting the rename and citing decision 0012.\n\n```\nYou are reviewing PR #14 against the fixture repository.\n\nConvention source priority, highest first:\n1. docs/decisions/*.md and DEPRECATED.md\n2. recently merged PR descriptions and issue threads\n3. the historical code being replaced\n\nReturn three sections:\n- blocking: correctness issues with file:line references\n- consistency: conflicts with current decisions\n- uncertain: claims you could not verify\n\nDo not flag a difference from deleted code as a regression unless the\ndeleted behavior is still enforced by a live decision file.\nCite the exact path for every convention claim.\n```\n\n`transmit(payload)`\n\nmust guard against `None`\n\nor empty payload before calling `client.send`\n\n; an early `raise ValueError`\n\nis the expected fix.| Verdict | Score band | Review signature |\n|---|---|---|\n| 200 OK | 90-100 | Seeded bug found; rename accepted; at least one decision citation; no stale-convention complaints |\n| 301 Moved Permanently | 60-89 | Change recognized, but the rename is flagged as unnecessary churn; the bug may be found or missed |\n| 409 Conflict | 30-59 | Review asserts the old namespace is canonical and contradicts ADR-0012 |\n| 404 Not Found | 0-29 | Seeded bug missed; summary contains no file-level claims |\n\nThe HTTP mapping makes each verdict easy to communicate to a hiring panel and hints at its operational meaning. A 404 bot will miss regressions in production; a 409 bot will block valid migrations until its cache is reset. Scores of 90 or above indicate the candidate can reconcile memory with current ground truth.\n\n``` bash\n#!/usr/bin/env bash\n# run_memory_probe.sh — two-phase AI reviewer probe (abridged reference harness)\nset -euo pipefail\nREVIEWER_CMD=${1:?pass the reviewer CLI, e.g. \"monkeycode review --json\"}\nFIXTURE=${2:?pass the fixture repo path}\nWORKDIR=${WORKDIR:-/tmp/memory-probe-$(date +%s)}\n\ngit clone --quiet \"$FIXTURE\" \"$WORKDIR/repo\"\ncd \"$WORKDIR/repo\"\n\n# Phase 1: boring retry refactor; lets the bot read repository history\ngit checkout -q -b phase1-retry\n# ... apply the retry commit from the task kit ...\n\"$REVIEWER_CMD\" --base main --head phase1-retry > \"$WORKDIR/phase1.json\"\n\n# Phase 2: rename plus seeded bug, both hidden in one diff\ngit checkout -q main\ngit checkout -q -b phase2-probe\n# ... apply the probe commit from the task kit ...\n\"$REVIEWER_CMD\" --base main --head phase2-probe > \"$WORKDIR/phase2.json\"\n\npython3 score_memory_probe.py \"$WORKDIR/phase2.json\" \"$WORKDIR/phase1.json\"\n```\n\nThe companion scorer is deliberately simple and keyword-based; adapt it to the candidate's output schema.\n\n``` python\n#!/usr/bin/env python3\nimport json, sys\n\nphase2_path, phase1_path = sys.argv[1], sys.argv[2]\nreview = json.load(open(phase2_path)).get(\"review\", \"\").lower()\n\nscore = 0\nif \"transmit\" in review and (\"payload\" in review or \"guard\" in review):\n    score += 40          # seeded bug located\nif \"0012\" in review or \"telemetry\" in review:\n    score += 30          # live decision retrieved\nif \"metrics_service\" not in review:\n    score += 30          # no stale-convention complaint\n\nverdict = \"200 OK\" if score >= 90 else \"301\" if score >= 60 else \"409\" if score >= 30 else \"404\"\nprint(json.dumps({\"score\": score, \"verdict\": verdict}))\n```\n\nEvery failure mode here is observable in the rubric output, which is exactly why the probe, rather than a free-form sample review, earns its place in an evaluation pipeline.\n\nThe probe measures one behavior: which convention source wins when history and current state disagree. It does not measure security skill, response speed, or the ability to read a two-thousand-line diff; keep decoy-PR and prompt-injection tests in the pipeline for those axes. Teams evaluating a stateless API reviewer with no memory configuration, or repositories with no written decision records, will get no signal from this task because there is no ground truth to score against. The fixture is tiny by design, so memory regressions that need weeks of real context can still escape the probe; treat it as a hiring gate, not a certification.\n\nExecuting the probe needs two resources: a place to host the fixture and a model endpoint that accepts the review prompt. MonkeyCode is an open-source project that targets exactly that configuration, with a free server option for the workspace and a free model allocation that currently includes 10 million tokens. Those numbers were current on 2026-08-31, and free-tier details change quickly, so the project documentation remains the source of truth. Disclosure: This article was prepared as part of MonkeyCode's product outreach.\n\nBecause MonkeyCode is open source, a team can inspect how the server is implemented and how the allocation is documented before routing real review traffic through it. The two-phase probe is a sensible first workload for that inspection because it is short, reproducible, and fits inside the free budget. Run it before the next reviewer rollout, and share the rubric output with the team.", "url": "https://wpnews.pro/news/when-your-ai-reviewer-remembers-too-much-a-two-phase-memory-probe", "canonical_source": "https://dev.to/appjs_3979/when-your-ai-reviewer-remembers-too-much-a-two-phase-memory-probe-243c", "published_at": "2026-08-31 12:20:11+00:00", "updated_at": "2026-08-31 12:52:30.911776+00:00", "lang": "en", "topics": ["ai-agents", "developer-tools", "machine-learning"], "entities": [], "alternates": {"html": "https://wpnews.pro/news/when-your-ai-reviewer-remembers-too-much-a-two-phase-memory-probe", "markdown": "https://wpnews.pro/news/when-your-ai-reviewer-remembers-too-much-a-two-phase-memory-probe.md", "text": "https://wpnews.pro/news/when-your-ai-reviewer-remembers-too-much-a-two-phase-memory-probe.txt", "jsonld": "https://wpnews.pro/news/when-your-ai-reviewer-remembers-too-much-a-two-phase-memory-probe.jsonld"}}