1Password's AI patching benchmark is misleading Trail of Bits reanalyzed 1Password's August 6, 2026 FLAWED report and found that 2,634 of 3,067 patches (86%) generated by 1Password's models blocked the supplied exploit in trials where agents could run code and were not instructed to apply the wrong fix, contradicting 1Password's headline 26% clean-fix rate. Trail of Bits said the 26% figure includes deliberately wrong-fix instructions, non-compiling patches, grading errors, and a handpicked sample, and warned that "defenders should not take 1Password's headline rate seriously as a measure of AI patching ability." Trail of Bits also reported that 283 of 2,265 first fixes (12.5%, one in eight) failed to fully resolve reported issues across 236 security assessments from 2024 to 2026, with a 95% confidence interval of 10.5% to 14.5%, and released two agent skills, post-patch-validation and review-walkthrough. 1Password’s FLAWED report https://1password.com/blog/why-ai-generated-patches-still-require-human-review , published on August 6, 2026, gives defenders a misleading picture of AI patching. Its headline says models produced clean fixes only 26% of the time. That figure includes experiments that deliberately instructed agents to apply the wrong fix, along with experiments in which agents could not compile or test their patches. The report risks making defenders less effective by discouraging them from using technology that could help them fix more vulnerabilities. Teams that take its headline at face value may leave repairable vulnerabilities unaddressed. We want our work to help defenders fix more vulnerabilities. This post shares real-world data on human and agent patch quality from our consulting projects and Patch the Planet. We’re also releasing two agent skills: post-patch-validation https://github.com/trailofbits/skills/tree/main/plugins/post-patch-validation to help agents test fixes, and review-walkthrough https://github.com/trailofbits/skills/tree/main/plugins/review-walkthrough to help engineers review them. Our review of 1Password’s code and data https://github.com/Off-by-1-Labs/FLAWED/tree/2d3d15693b155873709bcf0daa247c2f0221d694 found four choices that make its 26% clean-fix rate a misleading guide to ordinary patching work. 1 fn:1 1Password’s headline also obscures a useful result in its own data. We reanalyzed the patches and recorded test results published with the study, keeping trials where agents could run code and were not instructed to apply the wrong fix. In those trials, 2,634 of 3,067 patches generated by 1Password’s models 86% blocked the supplied exploit. We excluded runs that the study classified as having consulted the upstream fix. Blocking that exploit does not establish a complete repair, but these results show useful patching capability under reasonable working conditions that the headline fails to convey. The instructions and grading introduce further problems, several of which Davi Ottenheimer https://www.flyingpenguin.com/disinformation-pushed-by-1password-ai-patching-report-is-false/ has also highlighted: The grading errors can penalize valid fixes and let vulnerable patches pass. Combined with the handpicked sample and deliberately bad instructions, they leave the report without a credible basis for its headline. Defenders should not take 1Password’s headline rate seriously as a measure of AI patching ability. Understanding agent failures also requires understanding how often developers submit incomplete fixes. Our security consulting work gives us a detailed record of how developers repair vulnerabilities in their own software. We give clients detailed vulnerability reports, then conduct a “ fix review https://github.com/trailofbits/publications ” to check whether their proposed patches fully resolve the issues. Our records connect each vulnerability to the developer’s first proposed fix and our assessment of whether it worked. They preserve unsuccessful attempts that developers revise before an issue is considered resolved. We reviewed the first fixes submitted for 2,265 vulnerabilities across 236 Trail of Bits security assessments from 2024 to 2026. The developers maintained the affected software, had detailed reports from our engineers, and knew we would review their patches. Even under those favorable conditions, 283 first fixes failed to fully resolve the reported issue: 12.5%, or one in eight. Accounting for multiple fixes from the same assessment, the 95% confidence interval is 10.5% to 14.5% . Sometimes we point out a mistake in a client’s patch during an informal conversation, and they correct it before the formal fix review. Those early failures may never appear in the review record, so our data can undercount failed first attempts. We also excluded cases where the available records did not establish whether the fix worked. A direct comparison with agents would require the same tasks and working conditions. Through Patch the Planet, our joint initiative with OpenAI, Trail of Bits has co-authored hundreds of patches https://trailofbits.com/patch-the-planet/dashboard/ for widely used open-source projects. Agents wrote the patches with engineers directing the work and checking the results. Project maintainers then decided whether to merge, revise, or reject each submission. We examined the public review history of every Patch the Planet submission in our dataset that maintainers had merged or closed by September 14, 2026: 186 pull requests. 1Password’s benchmark used six vulnerabilities selected because their fixes were complex. Maintainers merged 126 of our 186 pull requests, an acceptance rate of 67.7%. 3 fn:3 In 91 of those 126 pull requests 72.2% , maintainers accepted the security fix we originally proposed. | Review outcome | PRs | % of merged PRs | |---|---|---| | Total merged | 126 | 100% | | Accepted with no security-relevant revision observed | 91 | 72.2% | | Accepted with security-relevant revision observed | 33 | 26.2% | | Indeterminate | 2 | 1.6% | Table 1: Changes requested by maintainers for 126 merged Patch the Planet pull requests. Security-related revisions include repairs to a proposed fix and expansions of its security coverage. Maintainer acceptance does not establish that every patch is correct. Maintainers closed the other 60 submissions without merging them. Most were superseded by other work or declined for policy, process, scope, or maintenance reasons. Four were explicitly rejected on technical grounds. | Reason for closure | PRs | % of closed PRs | |---|---|---| | Total closed without merge | 60 | 100% | | Superseded, reimplemented, or re-landed elsewhere | 36 | 60.0% | | Policy, process, scope, or maintenance reasons | 14 | 23.3% | | Duplicate or convergent with another fix | 3 | 5.0% | | Explicitly rejected on technical grounds | 4 | 6.7% | | Other or indeterminate | 3 | 5.0% | Table 2: Reasons maintainers closed 60 Patch the Planet pull requests without merging One of those closed submissions was our freenginx patch. 1Password’s case study examines a Patch the Planet fix for a memory-safety bug in freenginx’s embedded Perl module. An agent wrote our patch under the direction of a Trail of Bits engineer. It left one vulnerable code path open and introduced a new crash during request cleanup. The paper’s criticism of our patch is correct. The maintainer closed our pull request https://github.com/freenginx/nginx/pull/35 and committed a separate fix https://github.com/freenginx/nginx/commit/cf26435a029e75af7b0a2e8a26b1eefe445890c6 . That fix covered all three vulnerable code paths but introduced the same crash during cleanup. The paper documents the maintainer’s regression too. Both authors encountered the same trap. The original bug allowed Perl to destroy a callback before freenginx used it. Both fixes kept the callback alive so freenginx could use it later. But if the request timed out first, freenginx would make the request unusable and then release the callback. Releasing it could run Perl code that still tried to use the request, crashing the worker. Both authors missed a problem their fix could cause later, during cleanup. Catching it required looking beyond the original bug to what happened when a request ended early. Two authors, one human and one agent, working separately, made the same mistake on the same bug. Readers deciding whether to use agents need to know how their failures compare with those of human developers. Establishing which is more reliable requires measuring both under comparable conditions. We examined about 33,500 subsequent commits in Patch the Planet projects. When a later commit changed a file our patch had modified, we investigated whether it fixed a problem our patch had introduced. For each suspected regression, an agent attempted to demonstrate its impact with a proof of concept. Other agents and our engineers then challenged the findings. The review found at least ten functional bugs; four build, test, or release automation bugs; and one performance bug. It found no exploitable security vulnerabilities. Two examples illustrate the problems we identified: We are extending this investigation to every patch we authored, including patches with maintainer contributions. The findings will help us add checks that catch these failures before we submit future patches. We are releasing two agent skills alongside this post: post-patch-validation https://github.com/trailofbits/skills/tree/main/plugins/post-patch-validation to help agents test security fixes, and review-walkthrough https://github.com/trailofbits/skills/tree/main/plugins/review-walkthrough to help engineers review code changes. Post-patch-validation is a new skill we wrote to help agents catch incomplete fixes and regressions before submitting patches for review. It was not used in the Patch the Planet work described above. The skill starts with a vulnerability report and the code before and after the patch. It guides the agent through four tasks: Failed checks give the agent specific problems to investigate and repair before submitting its patch. The skill saves the tests and results so maintainers can see what was checked. To try post-patch-validation, install the skill https://github.com/trailofbits/skills/tree/main/plugins/post-patch-validation installation and give your agent the vulnerability report and the vulnerable and patched revisions: “Use post-patch-validation to validate the patch in HEAD against