Tests on typical inputs miss wrong LLM repairs; add one edge case

A preregistered arXiv study of 73 migration tasks built on 60 documented changes in CPython, Node.js and SQLite found that a release check running only typical inputs let through 0.410 of wrong LLM code repairs (task mean), while adding one author-written edge input drawn from the release note cut the wrong-release rate to 0.065 with correct repairs blocked unchanged at 0.010. The edge-input effect replicated on 15 test clusters of changes built after it was first observed, where the rate fell from 0.471 to 0.061, and held for both silent and explicit changes. The authors caution the rates describe how a release check behaves over a finite menu of 405 author-written documented fragments, not a rate for code a chat model writes, and the arXiv paper has not been peer reviewed.

The Short Answer With typical inputs only, the release check shipped a task mean wrong-released rate of 0.410 with correct blocked at 0.010 . For practitioners, the suggested shift is in what your gate runs: include edge inputs drawn from the release note rather than relying on typical inputs. This is an arXiv paper; we have not checked whether it has been peer reviewed. A code repair that runs cleanly can still return wrong results, and in a preregistered study the inputs of the pre-release check decided how many of those repairs shipped. With only typical inputs, a check that ran every requirement assertion released 0.410 of the wrong repairs task mean . Adding one edge input written from the release note cut that to 0.065, and false rejection of correct repairs stayed at 0.010. The study covered 73 migration tasks built on 60 documented changes in CPython, Node.js and SQLite. Every candidate repair ran on the real runtimes and was judged against contracts withheld from the repairer. The repairs were choices among author-written documented fragments; the free-form arm was not run. The rates therefore describe how a release check behaves over a finite menu of repairs. They are not a rate for code a chat model writes. Accepted but Wrong: Silent Failures of LLM Code Repairs after API Migrations https://arxiv.org/abs/2611.08269 is an arXiv paper; we have not checked whether it has been peer reviewed. The arXiv record notes: "30 pages, 2 figures. Preregistered; artifacts are available from the authors on request". For anyone gating LLM-repaired code, the practical result is a cheap step: add inputs drawn from the release note. Why a passing check does not mean the repair is right After a runtime or API upgrade, some breakages are loud, such as a removed function that raises an error. Others are quiet: every call succeeds and the result changes. The paper's examples are an enumeration that renders as 3 instead of Priority.HIGH , sum returning a more accurate float that no longer matches stored ledger totals, and Buffer.poolSize growing eightfold and changing how every upload is chunked. A repair can handle the loud kind and still fail the quiet kind. If your check only runs the typical input, a substantial share of wrong repairs gets released over the complete repair space. This is the old "overfitting" problem from automated program repair, applied to migrations: a patch passes the tests you have without being correct. What one edge input did to the release rate The authors compared sets of visible inputs under the same gate. They measured each set over every possible repair in each task's finite menu of documented fragments 405 fragments across all tasks . Here "wrong released" is the share of wrong procedures the gate let through, and "correct blocked" is the share of correct ones it rejected. Both are task means over the 52 affected test tasks. | Visible inputs | Wrong released | Correct blocked | |---|---|---| | Typical inputs only | 0.410 | 0.010 | | + one author-written edge input | 0.065 | 0.010 | | + 1 input from a blind LLM writer | 0.130 | 0.010 | | + up to 3 inputs from a blind LLM writer | 0.070 | 0.010 | | + up to 3 writer inputs, only those the writer flagged as "legacy-binding" | 0.157 | 0.010 | The edge-input effect replicated on the 15 test clusters of changes built after the effect was first seen. There the rate fell from 0.471 to 0.061. It also held for silent and explicit changes and with each runtime lineage left out in turn. What happened with real model repairs The authors then let four models pick repairs from the menu: GPT-5.6-luna , Qwen3-8B , gpt-oss-20b and qwen3.6-35b-a3b . The table shows counts out of 456 attempts per arm and model on the test partition, with the same all-assertions gate. | Model | Wrong releases, typical inputs | Wrong releases, + edge input | Correct completions, typical | Correct completions, + edge input | |---|---|---|---|---| | GPT-5.6-luna | 22 | 7 | 398 | 408 | | Qwen3-8B | 98 | 24 | 268 | 270 | | gpt-oss-20b | 71 | 8 | 320 | 353 | | qwen3.6-35b-a3b | 49 | 7 | 354 | 368 | In wrong releases per attempt, Qwen3-8B fell from 0.233 to 0.057, gpt-oss-20b from 0.181 to 0.020, and qwen3.6-35b-a3b from 0.120 to 0.018. The preregistered rules treated the first two as confirmatory and the other two as descriptive. For GPT-5.6-luna , which made few wrong releases anywhere, the drop from 22 to 7 was not statistically distinguishable Holm-adjusted p 0.499 . Why typical inputs miss the failure A wrong repair can match the old behavior on the common case On typical inputs, the unrepaired procedure failed visibly in only 34 of the 52 test tasks. An edge input written from the release note must make the stale procedure fail a visible check. In the paper's rules, the author's edge input had to make the stale procedure fail a visible check. That is why it reaches 52 of 52 tasks. If you write your own edge case, run that same test on it. The edge input narrows the gap but does not close it A later stage added 233 "hard distractors": wrong options that match the old output on typical inputs. The edge input still cut the overall rate to 0.058. But 0.168 of the hard wrong programs still passed, and the blind GPT-5.6-luna writer's inputs let through 0.153 of them. The edge input also cut wrong releases per attempt in all three open-weight families on this harder menu. The share of wrong programs a gate releases also depends on how the menu is built. On a larger menu of 1491 fragments, typical inputs released 0.278 of wrong programs and the edge input 0.045. Treat the headline rates as properties of the menu, not of the gate. Who writes the inputs matters Reviewers of the authors' earlier work objected that the edge inputs were author-written and close to the hidden inputs. So an isolated model, GPT-5.6-luna , acted as a blind writer, and it was also one of the repairers. It saw the requirement, the pinned excerpts, the typical input with its old output and the output checks, but never the program or hidden inputs, and made one call per task. Its inputs came close to the author's. Three of its 176 kept inputs equal a hidden input; without them, the rate with up to three inputs is 0.090. Two further results matter for practice: - The "which outputs still bind" label backfired. The writer was asked to flag inputs where the old output is still required, and only those flagged were kept. It labeled 35 of its 176 inputs as non-binding, and those were disproportionately the ones that revealed the change. The stale procedure failed visibly in 42 tasks under the guard, against 50 without it. - Weaker writers did not pass. The qwen3.6-35b-a3b writer raised false rejection to 0.083, beyond the preregistered 0.05 tolerance. The Qwen3-8B writer reached only 0.182 wrong released. The blind-writer result is specific to a frontier model. Self-tests, where the repairing model writes inputs for itself first, lowered wrong releases in every family for example 98 to 80 for Qwen3-8B and 71 to 47 for gpt-oss-20b . No reduction survived the correction for multiple comparisons. Smaller models wrote usable tests for fewer tasks: Qwen3-8B managed 30 of 73, GPT-5.6-luna 71. In a post-hoc comparison, gates built from the open-weight models' own unguarded tests let through more of those models' wrong first-step repairs than the blind frontier writer's gate did. A further post-hoc comparison points to writer strength rather than independence. On 21 shared tasks, same-model writer and repairer pairs averaged 0.161 wrong released and cross-model pairs 0.187. The authors call this a hypothesis, with four families and one prompt. What changes for you when an LLM fixes code after an upgrade The study did not test free-form code, so these steps are our reading of it. They apply the study's pattern to your own release checks. - Add at least one release-note input to every pre-release check. Pick an input that exercises the documented change, not another typical one. Confirm that the unrepaired code fails it visibly. - Have a separate, strong model draft two or three edge inputs. Give it the release note and the requirement, not your code or hidden checks. The best writer in the study was the frontier model, and weaker ones did not pass. - Compute expected outputs by running the old code on the old version. Never take them from the model. Drop inputs that error on the old version, plus duplicates and nondeterministic ones; those were the reasons the study discarded 27 writer inputs. - Keep every input the writer proposes. Do not let it decide which old outputs still apply. - Do not trust a weak model's tests for its own repair. Self-tests helped less than blind or author-written inputs, and a weaker model wrote usable tests for fewer tasks. - Expect a residual. Wrong repairs that match the old output on the typical input and also pass your edge input still exist, so keep human review for high-stakes changes. Limits: a finite menu, author-written tasks and one frontier writer - Repairs were picked from a menu. The authors planned a free-form arm, in which models write code that is then executed. The sandbox could not be built within the study machine's permissions, so the arm was not run, and no model-written code was executed. The paper does not claim the effects transfer to free-form repair. - The menus are author-written. Even the largest has fragments built by documented operators, and 25 tasks have fewer than 15 options. The authors also wrote the tasks and hidden inputs. - Three runtime lineages only. CPython, Node.js and SQLite are covered. The paper makes no claim about web APIs or other ecosystems. - GPT-5.6-luna played three roles. It was the blind writer, the contract auditor and one of the repairers. The writer's lead held on the other three families' repairs, but that is not a fully independent test. - The contracts were not reviewed by a human. Two blind AI audits flagged 34 of 73 contracts as stricter than their requirements, mostly through a file-preservation check. Dropping that check and legacy equality changed no verdicts. Disputes over whether the change class is right remain open. - The authors' own strong-evidence criterion was not met. The coverage-aware guard failed its preregistered test, and more than 10% of tasks were flagged by the AI audit. - The frontier model is hard to measure here. GPT-5.6-luna made so few wrong releases that its contrasts are descriptive. The gemma-4-12b-qat family was not admitted after 0/12 exact outputs, with answers that began with leaked reasoning text, which the paper calls a serving problem; the paper draws no conclusion about it. Try this prompt Use this in a fresh chat with your strongest model, before you gate a repaired function on the new version. Do not paste your code into it. You are writing test inputs for a migration check, not fixing code. Below are the release-note text for one change, the requirement the code must still meet, and one typical input with its keys and value types. You have not seen the code, so do not guess at it. Task: write up to three new inputs that exercise the documented change and its boundary values, so that code still relying on the old behavior would produce a visibly different result. Rules: - Each input must use exactly the same keys and value types as the typical input, be deterministic, and run without error on the OLD version. - Output only the inputs, as JSON, one object per input, each with a one-line reason tied to a sentence in the release note. - Do not give expected outputs. I will compute them by running the old code on the old version. - Do not decide which old outputs still need to hold. List every input you propose. - If the release note gives no basis for an edge input, say so instead of inventing one. RELEASE NOTE: PASTE THE RELEVANT TEXT REQUIREMENT: PASTE WHAT THE CODE MUST STILL DO TYPICAL INPUT: PASTE ONE EXAMPLE INPUT Adapt it to your own task with the Prompt Optimizer https://www.thepromptindex.com/prompt optimizer/ . Sources Used This article is a plain-English breakdown of the following paper. This is an arXiv paper; we have not checked whether it has been peer reviewed. Read the original for the full methodology and results: - Accepted but Wrong: Silent Failures of LLM Code Repairs after API Migrations https://arxiv.org/abs/2611.08269 arXiv:2611.08269 - Authors: Xiaobo Chen, Genliang Zhu, Ziyuan Wang and Chu Wang - The arXiv record notes: '30 pages, 2 figures. Preregistered; artifacts are available from the authors on request'. Frequently Asked Questions How does the release check behave with typical inputs versus adding an edge input? In the study’s gate, typical inputs only resulted in a wrong-released task mean of 0.410, with correct blocked at 0.010. Adding one author-written edge input from the release note reduced wrong released to 0.065 while correct blocked remained 0.010. Does adding an edge input increase false rejection of correct repairs? The study reports correct blocked stayed at 0.010 for both typical inputs only and typical inputs plus one release-note edge input. That indicates the false rejection rate for correct repairs did not increase between those two conditions. What is the main limitation to keep in mind when applying this in practice? The study does not test free-form code; any practical steps are an extrapolation from the study's pattern, not something it tested. The paper also frames rates as properties of the finite menu of repairs considered.