Recent discussions about AI assistants that trust every archived comment raise a practical question for engineering teams: does an AI code reviewer become more accurate when given the full repository history, or does it just become more confidently wrong? Experience with review bot evaluations suggests that an excessive or stale context often produces worse feedback than a bare diff. The following take-home task offers a repeatable method for separating useful contextual awareness from harmful memory, using only a small Python script and a free model endpoint.
Most AI code review tools advertise deep repository awareness, but nobody tests how that awareness degrades when the repository contains outdated TODOs or superseded architecture decisions. A reviewer that naively trusts every comment can reject a perfectly valid fix because a two-year-old note says otherwise. The test below builds a small repository with a deliberately misleading comment, then runs the same prompt under three context regimes. The result shows whether a candidate tool can ignore noise without losing signal.
The exercise is designed to be completed in under 90 minutes. Candidates receive a prompt, a sample pull request that fixes a real bug, and a rubric. They must run an AI reviewer against the PR using three context configurations and report scores. No proprietary infrastructure is required; ordinary laptops and free-tier APIs work fine.
The repository is a tiny Flask application with a calculate_discount
function. The PR changes the discount formula from a flat 10% to a tiered system based on order amount. It also adds a unit test and updates the README. Crucially, the code contains a stale comment in app.py
: # TODO: after the Black Friday sale, remove the flat 10% discount
— a comment that was accidentally left from a previous sprint and now contradicts the PR's intent.
The same prompt is used in all three runs, asking the reviewer to identify the functional change, check for regressions, and assess test coverage.
The exact prompt is shown below. It is deliberately neutral to avoid steering the model toward or away from the stale comment.
You are reviewing a pull request. Here is the diff:
<DIFF>
Focus on the functional behavior change introduced by the diff. Identify any bugs, regressions, missing edge cases, or test gaps. Do not comment on code style unless it directly affects correctness. Return a summary with a severity for each finding.
The rubric awards points across four dimensions, with a maximum score of 20:
A perfect reviewer scores 20. A tool that worships the stale comment scores low on dimension two, even if it nails the functional detection.
A well-written human review would mention the change from a flat 10% to a progressive 5/10/15% structure and recommend adding a test for orders at $100 and $200. It would also say that the TODO comment is now obsolete and should be removed in a follow-up cleanup. Finally, it would note that the new test covers normal and high-value orders but misses the boundary exactly at $100. This reference answer requires no deep repository archaeology; it relies only on the diff and a few seconds of inspection.
During informal runs against several open-weight models, three distinct failure patterns appeared. First, models with full git history tended to quote prior commits that referenced the old flat discount, treating them as current specifications. Second, models given the stale note often rejected the PR with a high severity finding, arguing that the change violates the documented requirement. Third, diff-only models occasionally missed the boundary test gap because they did not see the surrounding function that defines the thresholds. These failure modes are consistent with the broader observation that AI systems remember everything and trust all of it, but they rarely discriminate by timestamp or relevance.
MonkeyCode is an open source assistant platform that bundles a free server option and a substantial token allowance for model calls, currently advertised at ten million tokens for new accounts. This makes it a practical, zero-cost execution environment for the three-regime experiment. A developer can self-host the MonkeyCode server on any small VPS, send the prompt through its REST interface, and capture the scoring output without touching a paid API. The platform's model routing supports various open-weight checkpoints, though the exact list changes over time, so the documentation should be consulted for current availability. Disclosure: This article was prepared as part of MonkeyCode's product outreach.
Note: the token allowance and free server availability were verified from the project's README on the date of writing, but quotas and model selections may change without notice.
The following Python snippet automates the three runs against a local MonkeyCode server and writes the model responses to files for later scoring:
import requests
import json
server_url = "http://localhost:8000/v1/chat/completions"
prompt = "You are reviewing a pull request. Here is the diff:\n<DIFF>\nFocus on the functional behavior change introduced by the diff. Identify any bugs, regressions, missing edge cases, or test gaps. Do not comment on code style unless it directly affects correctness. Return a summary with a severity for each finding."
for mode, diff_file in [("full", "full_context.diff"), ("diff", "diff_only.diff"), ("stale", "diff_with_stale.diff")]:
with open(diff_file) as f:
user_content = prompt.replace("<DIFF>", f.read())
payload = {
"model": "local-model",
"messages": [{"role": "user", "content": user_content}]
}
r = requests.post(server_url, json=payload)
with open(f"response_{mode}.json", "w") as out:
json.dump(r.json(), out, indent=2)
The script assumes the three diff files are prepared separately, which keeps the experiment reproducible. The output JSON contains the raw model response, and a separate scoring sheet can be filled in manually or with a simple regex parser.
This experiment is intentionally narrow. It evaluates only contextual discipline for a single bug-fix PR, not the full breadth of code review competence. Teams should not hire or dismiss an AI reviewer based solely on this score. The test also ignores security review, multi-file architectural impact, and language-specific pitfalls. It is most useful as a quick filter for detecting reviewer candidates that treat every repository comment as gospel, a problem that tends to appear strongly in this simple scenario.
Teams that already use a diff-only reviewer with no repository memory will find the test less informative, since the stale-comment failure mode cannot occur. Similarly, engineering groups that enforce daily comment cleanup and never allow outdated TODOs to persist may see no difference between the three configurations. For those situations, a more relevant test would measure the tool's ability to use design documents or issue trackers as external context, which is a different exercise entirely.
The minimal-context test gives teams a concrete, five-minute artifact for spotting a specific but widespread weakness in AI reviewers: the inability to separate stale information from current requirements. A good reviewer should behave like a focused colleague who knows the diff, asks about the background, and refuses to follow a comment that history has proven wrong. Using MonkeyCode's free server and token allowance makes the experiment accessible to any team that wants evidence before wiring an AI reviewer into its pull request pipeline.