{"slug": "everyone-can-find-vulnerabilities-now-here-is-how-to-tell-them-apart", "title": "Everyone Can Find Vulnerabilities Now. Here Is How To Tell Them Apart.", "summary": "Wiz, Microsoft, Bugcrowd, Nebula Security, and Aikido all announced AI systems that find real software vulnerabilities, but benchmarks like CyberGym measure known-bug reproduction, not real-world discovery. Wiz's Project Atlas scored 90.9 percent and found over 200 unknown vulnerabilities, while Microsoft claimed 95.95 percent on the same benchmark. Buyers should ask vendors what the AI started from, what it can see, and how many findings were previously unknown.", "body_md": "# Everyone Can Find Vulnerabilities Now. Here Is How To Tell Them Apart.\n\nIn the space of about seventy-two hours this month, a cloud security giant, a hyperscaler, a bug bounty platform, and a Y Combinator startup all announced that their AI can find real vulnerabilities in real software.\n\nIf you buy security tooling, your inbox is about to get loud. Every one of those announcements uses the same three words: autonomous, validated, exploitable. They do not mean the same thing.\n\nSo here is the field guide. What actually happened, what it signals, and the three questions that separate the products from the press releases.\n\n## What happened\n\nWiz, now part of Google, announced Project Atlas, an agentic system for vulnerability research. It took the top spot on CyberGym with a 90.9 percent success rate and, per Wiz, has surfaced more than two hundred previously unknown vulnerabilities in open source code that has been fuzzed and reviewed for years. Days later Microsoft claimed 95.95 percent on the same benchmark with its own agent harness.\n\nBugcrowd launched Savant Pathseeker, an agentic pentesting product for web applications and APIs, currently in early access.\n\nNebula Security came out of Y Combinator with Vega, an AI agent built by members of a world champion CTF team, with a public catalog of over a thousand bugs found.\n\nAnd Aikido, having already acquired two autonomous pentesting companies and an AI code review company, bought another security startup outright.\n\nFour announcements. One conclusion, which we have been arguing for two years: AI can now find vulnerabilities that human experts find, at a speed and cost that human experts cannot match. That question is settled. Stop debating it.\n\nThe interesting question is the one nobody is answering in their launch post.\n\n## The crash test problem\n\nCar manufacturers publish crash test ratings. Five stars is meaningful. It is also a controlled collision, at a known speed, into a known barrier, with sensors already positioned where the engineers expect the impact.\n\nNobody confuses that with driving.\n\nBenchmarks work the same way. CyberGym, the benchmark everyone raced to the top of this month, hands a system a vulnerability that already exists. It provides the description. It provides the relevant code. Then it asks whether the system can produce a working proof of concept.\n\nThat is a genuinely useful measurement, and the teams competing on it are doing serious engineering. But Wiz says this themselves, to their credit, in the same post announcing their score: reproducing a known vulnerability is a different and easier problem than discovering an unknown one. Real discovery starts from a blank page. No description. No pointer to the vulnerable function. A codebase that changes every day.\n\nThat is the job. No public benchmark measures it yet. Which means the leaderboard is telling you something real about engineering quality and almost nothing about what the product will do to your codebase on a Tuesday.\n\nWe have chosen not to optimize for that leaderboard. Not because we would do badly on it, but because tuning a product to score well on known-bug reproduction is a decision to get better at the easy half of the problem. We would rather be measured on what we find in your code that nobody has ever found before.\n\n## Three questions worth asking any vendor\n\n**What did it start from?**\n\nThere is a wide gap between a system handed a vulnerability and asked to prove it, and a system handed a repository and asked what is wrong with it. Ask which one you are buying. Ask how many of the findings in the last customer report were previously unknown.\n\n**What can it actually see?**\n\nThis one splits the market cleanly, and most buyers miss it.\n\nSome of these products test your running application from the outside. They send traffic, watch responses, and reason about behavior. They cannot read your source code, because they do not have it.\n\nOthers read your source code. They can trace a data flow across forty files and three services. They cannot tell you whether the endpoint is reachable in production behind your WAF.\n\nBoth are legitimate. Neither is complete. An attacker does not respect that boundary, and the vulnerabilities that end careers usually live in the seam: a logic flaw that only makes sense when you can see the code and the running system at the same time.\n\nThat seam is the reason we build both. Xint Code reads the source. Xint Web attacks the live application. They are converging into one platform because the interesting findings are the ones that need both halves.\n\n**What does it hand you at the end?**\n\nThis is the question that actually determines whether you got value.\n\nA weakness is a pattern in code that could be dangerous. A vulnerability is a specific, exploitable path through your system, with the steps to reproduce it. Traditional SAST hands you thousands of the first and calls it coverage. Your engineers then spend their quarter deciding which ones are real.\n\nWe built Xint around a simple constraint: if we cannot show you how to trigger it, we do not report it. Reproduction steps are not a nice feature on the finding detail page. They are the proof that the finding is real, and they are the difference between a report your engineers act on and a report your engineers argue with.\n\nThat is also where the market is heading whether vendors like it or not. [Regulators have started asking for demonstrated exploitability](https://xint.io/blog/fedramp-flat-scan-proof-of-exploitability) rather than a scanner output and a signature. Point-in-time attestation is on its way out. If your tooling cannot produce evidence, you are going to feel that within a year.\n\n## What the incumbents are conceding\n\nRead Bugcrowd's Pathseeker FAQ carefully. In the middle of a launch announcement, they state that agentic pentesting is not suited to complex business logic flaws, exploit chaining, or zero-days, and that it cannot satisfy compliance requirements on its own. That is an honest description of a product designed to raise a floor, with human researchers sold separately to reach the ceiling.\n\nIt is a reasonable business. It is also a very different bet from ours.\n\nWe think the ceiling is the product. Business logic, chained exploits, and previously unknown vulnerabilities are not the residue left over for humans after the machine has done the easy parts. They are the whole reason anyone pays for offensive security. Theori's team has spent a decade at Pwn2Own and in DARPA research proving that; the engine is built by the people who do that work, not around them.\n\n## What to do with all this\n\nDo not buy the benchmark. Ask what the system found last month that nobody had found before, and ask to see the reproduction steps.\n\nDo not buy half the picture. Ask whether the vendor can see your source code and your running application, or only one of them, and ask what they think lives in the gap.\n\nAnd do not let anyone sell you volume. The number of findings is the easiest metric in this industry to inflate and the least correlated with whether you are actually safer.\n\nThere is a lot happening. Most of it is real. The question is not whether AI can find vulnerabilities anymore. It is who can prove the ones they find.\n\nIf you want to see what that looks like against your own code, we will show you.", "url": "https://wpnews.pro/news/everyone-can-find-vulnerabilities-now-here-is-how-to-tell-them-apart", "canonical_source": "https://xint.io/blog/find-software-vulnerabilities-how-to-tell", "published_at": "2026-07-29 16:49:53+00:00", "updated_at": "2026-07-29 17:24:55.977553+00:00", "lang": "en", "topics": ["artificial-intelligence", "ai-products", "ai-agents", "ai-safety"], "entities": ["Wiz", "Google", "Microsoft", "Bugcrowd", "Nebula Security", "Aikido", "Y Combinator", "CyberGym"], "alternates": {"html": "https://wpnews.pro/news/everyone-can-find-vulnerabilities-now-here-is-how-to-tell-them-apart", "markdown": "https://wpnews.pro/news/everyone-can-find-vulnerabilities-now-here-is-how-to-tell-them-apart.md", "text": "https://wpnews.pro/news/everyone-can-find-vulnerabilities-now-here-is-how-to-tell-them-apart.txt", "jsonld": "https://wpnews.pro/news/everyone-can-find-vulnerabilities-now-here-is-how-to-tell-them-apart.jsonld"}}