LLMs’ ability to find real vulnerabilities at scale has shifted the bottleneck from bug discovery to triage, and especially, remediation.
In response, AppSec vendors are promising fully automated remediation. The first question to ask is, “what exactly do you mean by that?” because auto-remediation, where the AI gets to deploy to production unsupervised, is not what any enterprise could want.
This is a technically complicated issue but our work in practical offensive security has taught us that in complex codebases and environments an autonomous agent should not be publishing code changes without human review (as even the most advanced tech giants have found out).
However we are able to vastly accelerate that process from days or weeks to often less than 30 minutes of human time per bug fix, including post-patch validation.
The promises being made #
The pitch sounds amazing: Everything is automated - from bug discovery to validation, and remediation.
But patches are also code that can introduce their own set of bugs and unintended consequences to complex environments. We also know AI-generated code is itself more likely to be flawed and error prone, compounding the security or performance issues.
In fact, a study by Off-by-1 Labs showed that 54% of complex patches generated by AI tended to have serious flaws. Generating a code snippet to patch a vulnerability seems easy, but in the real world we have seen it complicate product security teams’ lives at each step of the patching process.
How AI-generated code patches can fail #
There are three main failure modes to AI-generate patches:
The patch doesn’t do anything: It doesn’t fix the underlying issue but it also doesn’t break anything.Breaking change: The patch fixes the vulnerability but breaks the app appsNew vulns: The patch introduces new vulnerabilities even as it fixes the vuln it set to fix.
To even get a code snippet that reliably compiles correctly (never mind doesn’t cause unexpected downstream impacts), the autonomous AI AppSec platform would need to be able to test a fix. To test a fix, the AI system would need to be provided with a way to build and test and debug the target application, usually in some sort of dev container.
Getting this is much harder than it sounds, for many reasons, including:
Most developers depend on their CI, so building and maintaining a dev container for an outside vendor like Xint requires its own resources.
Note that building a thing and running tests often requires building many recursive dependencies for that thing.
Even if a dev container exists, these things very much tend to not “just work” in the way an automated system would need it to.
Creating and/or configuring each client’s dev container for the AI system would require manual work 90% of the time. Assuming this is executed correctly, there’s still going to be ongoing friction with stuff that a client has to resolve on their end.
Many codebases (especially non-release versions) contain failing tests. Often nondeterministically-failing ones. It’s especially hard to automatically tell whether a proposed remediation breaks a codebase that already appears broken.
Claude Code and Codex both sidestep some of these problems. They do not need a container, because they run live on the developer's machine, but that is exactly the difference: Xint, as an external service, would not be able to replicate the same approach as a dev machine that can run tests.
The fact that Claude Code and Codex work and avoid these problems doesn't undermine our argument; in fact, Claude Code and Codex are the type of human-in-the-loop patching that we think does work. It's very much not "fully automatic".
Xint’s Accelerated Remediation Solution #
Xint’s post scan results propose a patch alongside a description of exactly what the bug is and why it leads to a security vulnerability. The key steps of merging changes and testing those changes stays the responsibility of code owners.
After the fix has been installed, the code owner re-drives the original exploit against the running app on Xint Web and sees that the exploit now fails, which is then reflected in the report tied to that finding to show that the issue has been addressed. Xint Code fixes are similar: "this finding is closed, here's the proof." There’s no other provider that can do post-patching validation for both source code and runtime.
In our initial tests, ~80% of Xint’s AI-generated code snippets successfully patched the vulnerability without introducing new issues, and the remainder neither fixing nor introducing new issues. This puts us ahead of the Off-by-1 study’s benchmarks where 54% of patches fail.
We believe this is because the quality of Xint’s bug validation outputs is so high that it leads to better context for AI coding agents to generate working patches.
What we have chosen not to do #
At least for now, we purposefully did not add a button to our interface that automatically creates a pull request. This is a forcing function so that human engineers will, in the process of copying and then inserting the code snippet, have to review any diff. From there the process is similar where the patch is deployed and the engineer tests against running application to see if it compiles, with a Xint re-scan included to see if the vulnerability has been successfully patched and no new vulnerabilities introduced.
We have exciting iterations in the months ahead that will build on this capability, but for now this approach best accelerates the remediation bottleneck while maintaining engineers in the loop to avoid any catastrophic changes introduced by new code snippets.
Even though Xint is AI-native, we absolutely live by human-in-the-loop Whether or not an organization chooses to require human review is up to them. Human review is a "bottleneck" the same way quality assurance and engineering validation are bottlenecks. You can skip it, but it's a gamble every time you do. The high quantities of code LLMs produce are a breeding swamp for software bugs. If you're prioritizing development velocity over all else and okay with your code being full of unknown bugs, then absolutely, auto-generate code and ship it without review.
Products like Xint—that add AI review for bugs, security and otherwise— accelerate the finding and fixing bugs before they get deployed.
But many organizations have less tolerance for releasing and using buggy code. Xint is one of them. We demand a higher level of assurance that our software actually works and doesn't have security vulnerabilities. Today, this absolutely means we do both AI and human review of all our code.