{"slug": "i-seeded-bugs-into-my-own-pr-to-test-the-ai-reviewer", "title": "I Seeded Bugs Into My Own PR to Test the AI Reviewer", "summary": "A developer built a regression test harness to evaluate AI code reviewers by seeding known bugs into pull requests. The harness, which uses the OpenAI chat-completions API, checks whether the model flags the planted bug, providing a snapshot of reviewer performance that can be run on a schedule. The developer tested the harness with a simple function containing a flipped sign bug and reported that the model caught it.", "body_md": "I was one merge away from shipping. The AI reviewer had already spoken: \"No issues found. Looks good to me.\"\n\nAnd that's when the doubt hit. What does \"looks good\" mean when the reviewer is a model I've never tested?\n\nSo I stopped reviewing the code and started reviewing the reviewer. I planted a bug I already knew about, asked the model to find it, and scored the answer. The question was simple: can a free model reviewer catch a bug I already know is there? And if it can't, how would I ever find out?\n\nHere's the function I used. It's small, it's realistic, and it contains exactly one logic bug.\n\n``` python\ndef price_with_discount(price, discount_pct):\n    if discount_pct < 0 or discount_pct > 100:\n        raise ValueError(\"discount must be between 0 and 100\")\n    factor = 1 + discount_pct / 100  # seeded bug: should be minus\n    return round(price * factor, 2)\n```\n\nRead it once. If you said \"the sign is flipped,\" you just passed the test. The real question is whether a model reviewer says the same thing when this function is buried in a longer file.\n\nThere's a discussion on DEV this week about who reviews the reviewer now that AI writes the first draft. I'm the nobody in that discussion. I'm a student; my review process is reading my own code and hoping. A free model reviewer sounds like the upgrade I can actually afford — until I remember that hope is not a test plan.\n\nSo I built one. The goal was a reviewer regression test. Real software has regression tests, and a reviewer is software, so it deserves a test too. The test is brutally simple: seed a known bug, ask for a review, check whether the bug gets flagged. Run it once and you have a snapshot. Run it on a schedule and you have a trend — because models change, and a reviewer that worked last month might not work this month.\n\nPrerequisites: Python 3.10+, an endpoint that speaks the OpenAI chat-completions format, and an API key. I used the open-source project MonkeyCode's free model access for the endpoint and their free server option to host the scheduled run. Disclosure: This article was prepared as part of MonkeyCode's product outreach. The script itself doesn't care which provider you point it at.\n\nHere's the harness:\n\n``` python\n# review_harness.py\nimport json\nimport os\nimport sys\nimport urllib.request\n\nENDPOINT = os.environ[\"ENDPOINT\"]\nAPI_KEY = os.environ[\"API_KEY\"]\nMODEL = os.environ[\"MODEL\"]\n\n# (buggy_line, symptom) pairs — the ground truth you already know\nSEEDED_BUGS = [\n    (\"factor = 1 + discount_pct / 100\", \"increases the price\"),\n]\n\ndef load_code(path):\n    with open(path) as f:\n        return f.read()\n\ndef review(code):\n    payload = json.dumps({\n        \"model\": MODEL,\n        \"messages\": [{\n            \"role\": \"user\",\n            \"content\": (\n                \"You are reviewing a pull request. \"\n                \"Find logic bugs, not style nits. \"\n                \"For each bug, quote the line and explain the impact.\\n\\n\"\n                f\"```\n{% endraw %}\npython\\n{code}\\n\n{% raw %}\n```\"\n            ),\n        }],\n        \"temperature\": 0,\n    }).encode()\n    req = urllib.request.Request(ENDPOINT, data=payload, headers={\n        \"Content-Type\": \"application/json\",\n        \"Authorization\": f\"Bearer {API_KEY}\",\n    })\n    with urllib.request.urlopen(req) as resp:\n        data = json.load(resp)\n    return data[\"choices\"][0][\"message\"][\"content\"]\n\ndef main():\n    code = load_code(sys.argv[1])\n    response = review(code)\n    caught = any(symptom in response for _, symptom in SEEDED_BUGS)\n    print(json.dumps({\"caught\": caught, \"response\": response}))\n\nif __name__ == \"__main__\":\n    main()\n```\n\nRun it:\n\n```\nexport ENDPOINT=... API_KEY=... MODEL=...\npython review_harness.py price.py\n```\n\nExpected output — yours will differ in the response text, but the shape is the same:\n\n```\n{\"caught\": true, \"response\": \"Line 4: `factor = 1 + discount_pct / 100` — the sign is flipped. A 20% discount increases the price by 20% instead of reducing it.\"}\n```\n\nThe harness checks for a symptom phrase, not a perfect match. That's a deliberate tradeoff. It's easy to read, but it can also produce a false negative: if the model catches the bug and says \"the math is backwards\" instead of \"increases the price,\" my check misses it. The model was right and my test still failed. Remember that every time you see a green or red checkmark — it's a heuristic wearing a uniform.\n\nThe one-off run answers one question. The scheduled run answers a better one: is this reviewer still any good? I put the harness on MonkeyCode's free server and let cron do the rest.\n\n```\n0 9 * * * cd ~/reviewer-check && python3 review_harness.py price.py >> results.jsonl\n```\n\nEvery morning it appends one line to results.jsonl. After a week you have seven data points; after a month you have thirty. That's when the real insight shows up — not in any single review, but in the shape of the line. If cron's minimal environment can't find python3, replace it with the full path from `which python3`\n\n.\n\nThe run that made me keep the harness was a confident miss. The model said \"no logic errors found\" on a file that contained a planted sign flip. Nothing in the code had changed; the model had. That's the exact moment the setup earns its keep: confidence and accuracy are two different numbers, and only one of them was being measured before.\n\nThe exact numbers won't transfer to your setup. Your endpoint, your prompt, and your code will produce different ones. What transfers is the shape of the failure: a reviewer can be right on Monday and confidently wrong on Tuesday, and you won't know unless you're measuring.\n\nThree lessons stuck with me. First, a reviewer you can't test is just an opinion with better grammar. The harness gave me a number I could reason about, and a number beats a vibe. Second, seeding bugs is calibration, not sabotage. You're not trying to trick the model; you're trying to map its edge, and every red line is a point on that map. Third, a one-off test goes stale. Models change, prompts drift, and \"it worked last month\" is not evidence. The free server turned a snapshot into a time series, and the time series is what actually changed my behavior.\n\nNow the part nobody likes: who should not use this. If you're reviewing code you can't modify, this whole approach is out — you can't seed bugs into someone else's production PR. If you need a guarantee, this is still a heuristic; it measures one narrow capability, catching a planted logic bug, and it says nothing about architecture, security, or whether the model would have found the bug without a prompt that says \"find logic bugs.\" Free tiers also come with real constraints: rate limits, latency, and the occasional outage. The harness will happily log those as failures too, so read the raw responses before you blame the model.\n\nWhat should you take away? One sentence: an AI reviewer is software, and software needs a regression test. The test is a seeded bug, the assertion is a symptom phrase, and the schedule is a cron job. Build that before you trust any review — from a model, or from yourself.\n\nIf you want to run this exact setup, MonkeyCode's free model access and free server option are one way to get both pieces without a credit card. The value is in the harness, not the provider — point it at whatever endpoint you have and see what your reviewer is actually made of.", "url": "https://wpnews.pro/news/i-seeded-bugs-into-my-own-pr-to-test-the-ai-reviewer", "canonical_source": "https://dev.to/magickong/i-seeded-bugs-into-my-own-pr-to-test-the-ai-reviewer-92b", "published_at": "2026-08-26 10:46:10+00:00", "updated_at": "2026-08-26 11:15:27.856430+00:00", "lang": "en", "topics": ["ai-tools", "developer-tools", "artificial-intelligence"], "entities": ["MonkeyCode", "OpenAI"], "alternates": {"html": "https://wpnews.pro/news/i-seeded-bugs-into-my-own-pr-to-test-the-ai-reviewer", "markdown": "https://wpnews.pro/news/i-seeded-bugs-into-my-own-pr-to-test-the-ai-reviewer.md", "text": "https://wpnews.pro/news/i-seeded-bugs-into-my-own-pr-to-test-the-ai-reviewer.txt", "jsonld": "https://wpnews.pro/news/i-seeded-bugs-into-my-own-pr-to-test-the-ai-reviewer.jsonld"}}