1Password’s FLAWED report, published on August 6, 2026, gives defenders a misleading picture of AI patching. Its headline says models produced clean fixes only 26% of the time. That figure includes experiments that deliberately instructed agents to apply the wrong fix, along with experiments in which agents could not compile or test their patches.
The report risks making defenders less effective by discouraging them from using technology that could help them fix more vulnerabilities. Teams that take its headline at face value may leave repairable vulnerabilities unaddressed.
We want our work to help defenders fix more vulnerabilities. This post shares real-world data on human and agent patch quality from our consulting projects and Patch the Planet. We’re also releasing two agent skills: post-patch-validation to help agents test fixes, and review-walkthrough to help engineers review them.
Our review of 1Password’s code and data found four choices that make its 26% clean-fix rate a misleading guide to ordinary patching work.1
1Password’s headline also obscures a useful result in its own data. We reanalyzed the patches and recorded test results published with the study, keeping trials where agents could run code and were not instructed to apply the wrong fix. In those trials, 2,634 of 3,067 patches generated by 1Password’s models (86%) blocked the supplied exploit. We excluded runs that the study classified as having consulted the upstream fix. Blocking that exploit does not establish a complete repair, but these results show useful patching capability under reasonable working conditions that the headline fails to convey.
The instructions and grading introduce further problems, several of which Davi Ottenheimer has also highlighted:
The grading errors can penalize valid fixes and let vulnerable patches pass. Combined with the handpicked sample and deliberately bad instructions, they leave the report without a credible basis for its headline. Defenders should not take 1Password’s headline rate seriously as a measure of AI patching ability.
Understanding agent failures also requires understanding how often developers submit incomplete fixes. Our security consulting work gives us a detailed record of how developers repair vulnerabilities in their own software. We give clients detailed vulnerability reports, then conduct a “fix review” to check whether their proposed patches fully resolve the issues.
Our records connect each vulnerability to the developer’s first proposed fix and our assessment of whether it worked. They preserve unsuccessful attempts that developers revise before an issue is considered resolved.
We reviewed the first fixes submitted for 2,265 vulnerabilities across 236 Trail of Bits security assessments from 2024 to 2026. The developers maintained the affected software, had detailed reports from our engineers, and knew we would review their patches. Even under those favorable conditions, 283 first fixes failed to fully resolve the reported issue: 12.5%, or one in eight.
Accounting for multiple fixes from the same assessment, the 95% confidence interval is 10.5% to 14.5%. Sometimes we point out a mistake in a client’s patch during an informal conversation, and they correct it before the formal fix review. Those early failures may never appear in the review record, so our data can undercount failed first attempts. We also excluded cases where the available records did not establish whether the fix worked. A direct comparison with agents would require the same tasks and working conditions.
Through Patch the Planet, our joint initiative with OpenAI, Trail of Bits has co-authored hundreds of patches for widely used open-source projects. Agents wrote the patches with engineers directing the work and checking the results. Project maintainers then decided whether to merge, revise, or reject each submission.
We examined the public review history of every Patch the Planet submission in our dataset that maintainers had merged or closed by September 14, 2026: 186 pull requests. 1Password’s benchmark used six vulnerabilities selected because their fixes were complex.
Maintainers merged 126 of our 186 pull requests, an acceptance rate of 67.7%.3In 91 of those 126 pull requests (72.2%), maintainers accepted the security fix we originally proposed.
| Review outcome | PRs | % of merged PRs |
|---|---|---|
| Total merged | 126 | 100% |
| Accepted with no security-relevant revision observed | 91 | 72.2% |
| Accepted with security-relevant revision observed | 33 | 26.2% |
| Indeterminate | 2 | 1.6% |
Table 1: Changes requested by maintainers for 126 merged Patch the Planet pull requests. Security-related revisions include repairs to a proposed fix and expansions of its security coverage.
Maintainer acceptance does not establish that every patch is correct.
Maintainers closed the other 60 submissions without merging them. Most were superseded by other work or declined for policy, process, scope, or maintenance reasons. Four were explicitly rejected on technical grounds.
| Reason for closure | PRs | % of closed PRs |
|---|---|---|
| Total closed without merge | 60 | 100% |
| Superseded, reimplemented, or re-landed elsewhere | 36 | 60.0% |
| Policy, process, scope, or maintenance reasons | 14 | 23.3% |
| Duplicate or convergent with another fix | 3 | 5.0% |
| Explicitly rejected on technical grounds | 4 | 6.7% |
| Other or indeterminate | 3 | 5.0% |
Table 2: Reasons maintainers closed 60 Patch the Planet pull requests without merging
One of those closed submissions was our freenginx patch.
1Password’s case study examines a Patch the Planet fix for a memory-safety bug in freenginx’s embedded Perl module. An agent wrote our patch under the direction of a Trail of Bits engineer. It left one vulnerable code path open and introduced a new crash during request cleanup. The paper’s criticism of our patch is correct.
The maintainer closed our pull request and committed a separate fix. That fix covered all three vulnerable code paths but introduced the same crash during cleanup. The paper documents the maintainer’s regression too.
Both authors encountered the same trap. The original bug allowed Perl to destroy a callback before freenginx used it.
Both fixes kept the callback alive so freenginx could use it later. But if the request timed out first, freenginx would make the request unusable and then release the callback. Releasing it could run Perl code that still tried to use the request, crashing the worker. Both authors missed a problem their fix could cause later, during cleanup. Catching it required looking beyond the original bug to what happened when a request ended early.
Two authors, one human and one agent, working separately, made the same mistake on the same bug. Readers deciding whether to use agents need to know how their failures compare with those of human developers. Establishing which is more reliable requires measuring both under comparable conditions.
We examined about 33,500 subsequent commits in Patch the Planet projects. When a later commit changed a file our patch had modified, we investigated whether it fixed a problem our patch had introduced. For each suspected regression, an agent attempted to demonstrate its impact with a proof of concept. Other agents and our engineers then challenged the findings.
The review found at least ten functional bugs; four build, test, or release automation bugs; and one performance bug. It found no exploitable security vulnerabilities. Two examples illustrate the problems we identified:
We are extending this investigation to every patch we authored, including patches with maintainer contributions. The findings will help us add checks that catch these failures before we submit future patches.
We are releasing two agent skills alongside this post: post-patch-validation to help agents test security fixes, and review-walkthrough to help engineers review code changes.
Post-patch-validation is a new skill we wrote to help agents catch incomplete fixes and regressions before submitting patches for review. It was not used in the Patch the Planet work described above.
The skill starts with a vulnerability report and the code before and after the patch. It guides the agent through four tasks:
Failed checks give the agent specific problems to investigate and repair before submitting its patch. The skill saves the tests and results so maintainers can see what was checked.
To try post-patch-validation, install the skill and give your agent the vulnerability report and the vulnerable and patched revisions:
“Use post-patch-validation to validate the patch in HEAD against <vulnerable-commit>, using the vulnerability report in <report-path>.”
Review-walkthrough helps engineers review the patches they are responsible for merging. It turns a branch’s complete diff into an interactive walkthrough that explains the changes in a logical reading order. Review findings appear beside the relevant code, where engineers can inspect them and draft their own comments. The walkthrough can also prepare a GitHub review for submission. Follow the quick start to generate a walkthrough for your branch.
These releases join our other public agent skills for improving security patches:
We publish these methods so other teams can use them to examine and improve their own patches.
A useful patching benchmark should measure whether agents help developers produce correct fixes and how much review those fixes require. The principles in our 2018 guide to evaluating fuzzing research apply here:
We are optimistic about AI’s usefulness to defenders. Through Patch the Planet, we are committing engineering time to fixing vulnerabilities alongside the people who maintain the affected software. We examine failures so we can improve our methods.
We will keep putting agents to work on difficult security problems and making the tools and lessons public. We want other teams to test our conclusions and take these methods further. Our goal is to give maintainers without dedicated security teams the ability to find and fix vulnerabilities that would otherwise go unaddressed.
The paper defines a clean fix as fully resolving the vulnerability without materially changing application behavior. ↩︎
The six-target mean has a standard error of about nine percentage points, which the report does not disclose. ↩︎
As of September 14, 2026, our dataset contained 240 public upstream pull requests. We excluded the 54 submissions still open from the outcome analysis. We count pull requests, each of which can contain more than one patch, and exclude maintainer-written replacements from our merged total. ↩︎