{"slug": "ai-generated-tests-can-make-coding-agents-worse-here-s-how-to-check-yours", "title": "AI-Generated Tests Can Make Coding Agents Worse. Here's How to Check Yours", "summary": "A developer demonstrated that AI-generated tests can degrade coding-agent repair performance, citing an ExecCritic preprint by Leitian Tao and colleagues in which weaker Qwen Test agent feedback dropped Qwen-3.5-35B-A3B's SWE-bench Verified resolve rate from 61.2% to 57.3%, while GPT-5.6-sol tests raised it to 65.3%. The writeup uses a Python order-filter example to show how a falsey-value bug fix can pass generated assertions while introducing a regression, and how a contradictory assertion can steer an automated repair loop toward damaging correct behavior.", "body_md": "*Originally published on [sergei-parfenov.com](https://sergei-parfenov.com/blog/ai-generated-tests-can-make-coding-agents-worse/).*\n\nA bug fix can make every new test pass and still introduce a regression. Here is a deliberately constructed Python example, checked locally without an LLM.\n\nAn order filter has three requirements:\n\n`None`: return all orders.\nThe reported bug is that omitting the filter returns nothing. This proposed fix looks reasonable:\n\n```\nORDERS = [\n    {\"id\": 1, \"status\": \"paid\"},\n    {\"id\": 2, \"status\": \"pending\"},\n]\n\ndef filter_orders(orders, statuses=None):\n    if not statuses:\n        return list(orders)\n    return [order for order in orders if order[\"status\"] in statuses]\n\nassert filter_orders(ORDERS) == ORDERS\nassert filter_orders(ORDERS, [\"paid\"]) == [ORDERS[0]]\nprint(\"2 checks passed\")\n```\n\nBoth checks pass. Both branches of the `if` have been exercised. The reported symptom is fixed.\n\nNow add the check for the second requirement:\n\n```\nassert filter_orders(ORDERS, []) == []\n```\n\nIt fails. The function returns every order.\n\nPython treats both `None` and `[]` as falsey. Our requirements give them different meanings, and the patch erases that distinction.\n\nThe connection to coding agents becomes more consequential when those checks determine what the agent does next.\n\nOn September 8, Leitian Tao and colleagues published the [ExecCritic preprint](https://arxiv.org/html/2609.09133v1). Holding the Qwen-3.5-35B-A3B Repair agent fixed, they reported these SWE-bench Verified results:\n\n| Feedback source | Tasks resolved | \n|---|---|\n| Initial repair, before generated-test feedback | 61.2% | \n| Tests from the base Qwen Test agent | 57.3% | \n| Tests from GPT-5.6-sol | 65.3% | \n\nThe weaker tests reduced the resolved rate by **3.9 percentage points**. Better tests improved it.\n\nRates average three repair runs, reusing generated tests. Failed test qualification retains the initial patch in the all-task score. The baseline does not forbid repository tests. A separate official evaluator determines resolution. Feedback adds test-generation and revision work; compute budgets are not matched. These are the authors' results, not a benchmark replication for this article. [Method and results](https://arxiv.org/html/2609.09133v1#S5).\n\nA bad test can do more than miss a defect. It can give the next edit the wrong target.\n\nImagine adding this assertion to the filter example:\n\n```\n# This expectation contradicts the stated empty-list requirement.\nassert filter_orders(ORDERS, []) == ORDERS\n```\n\nOur broken patch passes it. A correct implementation would fail it. Feed that failure into an automatic repair loop, and the loop now has a reason to damage correct behavior.\n\nAdding assertions has strengthened the wrong interpretation.\n\nEven the familiar “fails before the fix, passes afterward” check needs a closer look. Here is the original implementation from the fixture:\n\n``` python\ndef filter_orders(orders, statuses=None):\n    statuses = statuses or []\n    return [order for order in orders if order[\"status\"] in statuses]\n```\n\nThe default-filter assertion fails against this version and passes against our proposed patch. It correctly detects the original bug. It simply cannot detect the new one.\n\nThe complete fix handles `None` explicitly:\n\n``` python\ndef filter_orders(orders, statuses=None):\n    if statuses is None:\n        return list(orders)\n    return [order for order in orders if order[\"status\"] in statuses]\n```\n\nRunning the same checks against all three implementations produces:\n\n| Implementation | Default + paid-filter checks | Those checks + empty-list check | \n|---|---|---|\n| Original | 1 passes, 1 fails | 2 pass, 1 fails | \n| Plausible patch | 2 pass | 2 pass, 1 fails | \n| Corrected patch | 2 pass | 3 pass | \n\nThe [runnable companion](https://sergei-parfenov.com/assets/downloads/ai-test-demo.zip) includes all three implementations, the checks, and the verified output. It uses Python’s standard library and makes no LLM or network calls.\n\nThe extra check earns its place because it distinguishes two implementations the earlier checks considered equally acceptable.\n\nThat is the question I would bring to an AI-generated test review: **which plausible wrong implementation would this test reject?**\n\nFor the filter, the candidate mistakes are easy to name: ignore the filter entirely, treat every missing filter as empty, or treat every empty filter as missing. They correspond to different misunderstandings of the contract. Tests that separate those cases tell us more than several additional examples of paid orders.\n\nThis is also where [mutation testing](https://mutmut.readthedocs.io/en/latest/) can help: make small changes to the implementation and check whether the suite detects them. Inspect surviving mutations to understand what they change; some are equivalent for the supported inputs. For this fixture, changing `statuses is None` to `not statuses` is a useful manual mutation because it has a known, observable effect on required behavior.\n\nFor an agent workflow, I would make four changes.\n\n**Write down the expected behavior before reviewing the patch.** Include the ordinary case, the reported failure, and the neighboring case most likely to be confused with it. Here, `None` and `[]` belong on separate rows. If the issue leaves that distinction unspecified, get a product decision before turning either interpretation into a test.\n\n**Review expected values as carefully as production code.** An assertion is a claim about the product. Trace that claim to a requirement, an established compatibility promise, or an independently checked example. Copying the current output into an expected value can preserve the exact behavior you meant to question.\n\n**Keep an accepted regression check stable during repair.** Let the agent change the implementation while a separate runner evaluates it with the reviewed tests. Protect the test command and configuration too: an unchanged test file helps little if the patch can skip its execution. If the test itself is wrong, revise and review it explicitly, then evaluate the candidate again.\n\n**Inspect the failure before asking the agent to fix it.** An assertion showing the wrong returned orders is actionable behavioral evidence. A missing dependency, an import failure, or a command that selected zero tests needs a different response. Record what ran and why it failed.\n\nExecCritic separates test creation from repair, qualifies tests on the original repository, and keeps them unchanged during revision. That limits the repairer's ability to change its target. Separate contexts and permissions still cannot guarantee that both agents understood the issue correctly. [Paper](https://arxiv.org/html/2609.09133v1#S2); [released implementation](https://github.com/MSR-Orchard/execcritic).\n\nThe four steps above are a review procedure you can try in an existing project. They do not require training a model. Their value should be judged by the bugs and mistaken expectations they expose.\n\nFor your next AI-assisted fix, keep the original code, the proposed patch, and the new tests. Identify one plausible alternative implementation that violates the requirement. Run the tests against it.\n\nIf both implementations get the same green result, you have found a specific question the suite still cannot answer. Add the check that separates them, and review its expected result before trusting the next repair.", "url": "https://wpnews.pro/news/ai-generated-tests-can-make-coding-agents-worse-here-s-how-to-check-yours", "canonical_source": "https://dev.to/p0rt/ai-generated-tests-can-make-coding-agents-worse-heres-how-to-check-yours-3jc9", "published_at": "2026-09-11 11:57:46+00:00", "updated_at": "2026-09-11 12:10:38.872681+00:00", "lang": "en", "topics": ["ai-agents", "ai-research", "large-language-models", "developer-tools"], "entities": ["Leitian Tao", "ExecCritic", "Qwen-3.5-35B-A3B", "GPT-5.6-sol", "SWE-bench Verified", "Qwen Test agent", "sergei-parfenov.com"], "alternates": {"html": "https://wpnews.pro/news/ai-generated-tests-can-make-coding-agents-worse-here-s-how-to-check-yours", "markdown": "https://wpnews.pro/news/ai-generated-tests-can-make-coding-agents-worse-here-s-how-to-check-yours.md", "text": "https://wpnews.pro/news/ai-generated-tests-can-make-coding-agents-worse-here-s-how-to-check-yours.txt", "jsonld": "https://wpnews.pro/news/ai-generated-tests-can-make-coding-agents-worse-here-s-how-to-check-yours.jsonld"}}