Your AI agent fixed the bug, but what else did it change?

Trail of Bits published two open-source agent plugins, patching-bugs and regression-hunt, that delegate patch development and regression checking to AI agents, and reported that regression-hunt produced 25% fewer false-positive findings than a baseline Codex agent prompted to "review the PR for regressions" (95% confidence interval of 14% to 38%). The plugins grew out of Patch the Planet, Trail of Bits' joint initiative with OpenAI, and draw on a review of every patch a Trail of Bits engineer authored and maintainers merged upstream, which confirmed 21 regressions from over 130 merged PRs: 16 functional, one performance, and four build/test, with none breaking a security protection. The 21 confirmed regressions were identified among 58 candidates in a single run, roughly 36% precision, which Trail of Bits cited as evidence the industry needs to raise its signal-to-noise ratio as bug disclosures arrive at unprecedented speed and scale.

Today we’re publishing patching-bugs https://github.com/trailofbits/skills/tree/main/plugins/patching-bugs and regression-hunt https://github.com/trailofbits/skills/tree/main/plugins/regression-hunt , two plugins that let engineers delegate patch development and regression checks to agents. Our patching-bugs plugin reproduces reported bugs, develops fixes, and tests them. The regression-hunt plugin checks whether any changes break previously working and supported behavior. Engineers can direct the work and review their results; maintainers can supply their own security assumptions and patching requirements, so the agents work within the project’s rules. We built patching-bugs and regression-hunt through Patch the Planet https://trailofbits.com/patch-the-planet/ , our joint initiative with OpenAI, where we’ve co-authored hundreds of security patches for widely used open-source projects. Motivated by our major successes on projects like Scapy https://github.com/trailofbits/publications/blob/master/presentations/Auditing%20Scapy%20With%20Agents/README.md , our plugins draw on our analysis of merged patches and thousands of our engineers’ Codex sessions. The plugins provide maintainers with a structured workflow, evidence, independent checks, and best practices to improve their patching hygiene. In our preliminary evaluations, regression-hunt produces 25% fewer false-positive findings than a baseline Codex agent prompted to “review the PR for regressions.” Our September post https://blog.trailofbits.com/2026/09/15/1passwords-ai-patching-benchmark-is-misleading/ reported regressions introduced by some of our Patch the Planet patches. Since then, we’ve reviewed every patch that a Trail of Bits engineer authored and maintainers merged upstream. The review confirmed 21 regressions from over 130 PRs merged: 16 functional, one performance, and four build/test see appendix . None broke a security protection. “Regression” refers to a new defect introduced by a change that breaks previously correct and supported behavior along a path a real user or attacker can reach. For each patch, an agent looked for potential regressions, confirmed each one with a proof-of-concept PoC run before and after the change, and then passed the results to an engineer for review. The review was noisy: today’s bug disclosures arrive at an unprecedented speed and scale, and for patching to keep pace, the industry needs to increase its signal-to-noise ratio. The 21 confirmed regressions were identified among 58 candidates in a single run ~36% precision . But because agents take many different approaches and reach different results, each run generates more candidates to manually sort through. That led us to two conclusions: Standardizing both steps also makes every run repeatable and comparable. No fixed checklist will catch every regression, so we collect agent trajectories from our engineers’ patching sessions, and use the tasks that an earlier version failed to benchmark for the next one figure 1 . We used two primary sources to guide the design of our plugins: a review of every patch we merged upstream and the thousands of Codex sessions our engineers used to write those patches. This led to four principles that shaped our patching-bugs and regression-hunt plugins: The regression-hunt plugin Figure 2 : In addition to the four principles above, there are two core aspects that are critical to the plugin: Our review of merged PRs yielded a set of 21 confirmed regressions that we could use to evaluate regression-hunt. We used the whole set to compare the plugin with a baseline Codex agent prompted to “review the PR for regressions.” Across those runs, regression-hunt produced 25% fewer false-positive findings than the baseline, with a 95% confidence interval of 14% to 38%. Its strict evidence requirements also left some known regressions classified as borderline: the plugin identified a possible regression but did not gather enough evidence to confirm it. Fewer false positives therefore came with a tradeoff in how many known regressions it could confidently report. Next, we’ll investigate why these cases remained borderline and improve the plugin’s strategies for gathering evidence. Our goal is to confirm more real regressions while giving maintainers clearer information when the evidence remains inconclusive. The patching-bugs plugin is a distillation of all the best practices we’ve identified by analyzing thousands of Codex sessions by Trail of Bits engineers Figure 3 . In addition to the more traditional capabilities such as reproduction on live deployments and independent adversarial review , the plugin emphasizes: Both patching-bugs and regression-hunt will continue to evolve. Our engineers are using them in their Patch the Planet work now, and we’re collecting data on how they’re being used, what works, and what doesn’t. That data, along with better evaluations, will shape the next version’s capabilities: These plugins encode best practices we’ve found for an agent to patch a bug properly, and they improve when you tell them how your project works. The plugins work best when you input two kinds of project knowledge: THREAT MODEL.md PATCHING GUIDELINES.md or AGENTS.md For our Patch the Planet work with GPAC, our engineers wrote a threat model and a project-specific patching skill https://github.com/gpac/gpac/pull/3953 that captures how GPAC’s maintainers want bugs reproduced, what evidence a fix needs, and what a patch must preserve. Liboqs https://github.com/open-quantum-safe/liboqs/pull/2477 and validator.js https://github.com/validatorjs/validator.js/ have also published threat models in accordance with our guidance. The more comprehensive and specific these files are to your project, code conventions, and operating style, the better the output you’ll receive. As AI accelerates bug discovery, we need to turn those findings into reliable fixes just as quickly. Solutions need to balance speed and quality: an agent needs to understand a report, decide what needs to change, and check that a patch fixes the issue without introducing regressions. Our patching-bugs and regression-hunt plugins bring what we’ve learned from Patch the Planet into that workflow, helping maintainers move from a bug report to a patch they can confidently review and ship. Install patching-bugs https://github.com/trailofbits/skills/tree/main/plugins/patching-bugs and regression-hunt https://github.com/trailofbits/skills/tree/main/plugins/regression-hunt , add a THREAT MODEL.md and a PATCHING GUIDELINES.md to your repository, and put them to work on your next bug report. Give the tools your project’s context so they can help you focus on the right fix and validate it faster. Then share your feedback https://github.com/trailofbits/skills/issues , including comments about where our defaults don’t fit your project. Those real-world cases will help us build the next version. This table lists confirmed regressions in our patches, with links and a brief description of what broke. | Project | Our patch | Regression type | What regressed | |---|---|---|---| | botan | PR 5825 https://github.com/randombit/botan/pull/5825 | Functional | OCSP checking rejects valid responses signed by an intermediate issuer’s authorized delegate | | crypto-bigint | PR 1274 https://github.com/RustCrypto/crypto-bigint/pull/1274 | Build/test | DER-only unit tests fail to compile | | nats-server | PR 8276 https://github.com/nats-io/nats-server/pull/8276 | Functional | Refreshing inherited permissions rejects outstanding authorized replies | | nats-server | PR 8276 https://github.com/nats-io/nats-server/pull/8276 | Functional | A subscription deny briefly permits delivery of forbidden messages during default-permission updates | | noble-curves | PR 249 https://github.com/paulmillr/noble-curves/pull/249 | Functional | Existing custom BLS instances change verification behavior after callers mutate decoder callbacks | | noble-curves | PR 251 https://github.com/paulmillr/noble-curves/pull/251 | Performance | Valid recovered-signature verification repeats expensive elliptic-curve work | | noble-curves | PR 250 https://github.com/paulmillr/noble-curves/pull/250 | Functional | FROST DKG retries accept substituted round-one commitments without reauthentication | | pipx | PR 1827 https://github.com/pypa/pipx/pull/1827 | Build/test | The pre-release workflow cannot push Git changes without checkout credentials | | release-tools | PR 403 https://github.com/python/release-tools/pull/403 | Build/test | Release rollback cannot remove partial listing rows after failure | | scapy | PR 5099 https://github.com/secdev/scapy/pull/5099 | Functional | Editing a learned ECU response leaves lookup results stale | | scapy | PR 5100 https://github.com/secdev/scapy/pull/5100 | Functional | Deserializing a HICP configure message changes a specific target to broadcast | | scapy | PR 5101 https://github.com/secdev/scapy/pull/5101 | Functional | LDAP reassembly fails to emit a complete bind message after a late TCP prefix | | scapy | PR 5112 https://github.com/secdev/scapy/pull/5112 | Functional | Backward TCP reassembly drops payloads whose actual gap is within the limit | | simplex-chat | PR 7082 https://github.com/simplex-chat/simplex-chat/pull/7082 | Functional | Desktop portrait attachment previews display below their intended resolution | | simplexmq | PR 1804 https://github.com/simplex-chat/simplexmq/pull/1804 | Build/test | The cloud-script CLI test fails in generated source packages | | urllib3 | PR 5079 https://github.com/urllib3/urllib3/pull/5079 | Functional | Repeated IPv6 zone normalization makes the proxy destination disagree with the Host header | | urllib3 | PR 5095 https://github.com/urllib3/urllib3/pull/5095 | Functional | Valid unbracketed scoped IPv6 addresses are rejected or changed | | validator.js | PR 2822 https://github.com/validatorjs/validator.js/pull/2822 | Functional | Email validation accepts domain labels containing unpaired UTF-16 surrogates | | vllm | PR 51447 https://github.com/vllm-project/vllm/pull/51447 | Functional | Invalid Anthropic stop lists return HTTP 500 instead of a validation error | | vllm | PR 51446 https://github.com/vllm-project/vllm/pull/51446 | Functional | Separate-tokenizer chat templates are fetched at the wrong revision | | vllm | PR 51446 https://github.com/vllm-project/vllm/pull/51446 | Functional | FunAudioChat speech assets are fetched at the wrong revision |