AWS puts AI vulnerability detection to the test, and false positives pile up

AWS released the Deception Benchmark, a public dataset and evaluation process that measures how well AI models distinguish genuine security vulnerabilities from code that looks risky but is safe, and the results show false positives piling up. AWS said it is making the benchmark publicly available so researchers can use the dataset and evaluation process without repeating the cost of generating and refining the samples. High false-positive rates in AI-assisted vulnerability triage, penetration testing, threat modeling, incident response, and code review can create more work, increase alert fatigue, and reduce confidence in the tools, according to Help Net Security.