{"slug": "ai-security-scanning-needs-evidence-not-just-more-agents", "title": "AI Security Scanning Needs Evidence, Not Just More Agents", "summary": "A developer argues that Google's Mantis agentic vulnerability scanner is notable not for its \"agentic\" label but for its grounding structure — repository context, threat models, reviewer and critic stages, sandboxed reproduction, and evidence-tied patches. The writeup contends that vulnerability detection is several distinct jobs, and that AI security tools should surface working reproducers and residual uncertainty rather than model confidence, while spending reasoning budget only where it is needed.", "body_md": "Google’s Mantis caught my attention because it points to a problem most AI security demos quietly walk around: `finding a vulnerability is not the same as proving one exists.` \n\nThat distinction matters. Security teams already live with noisy scanners, half-useful alerts, and findings that require someone experienced to separate a real exploit path from a theoretical complaint. Adding an LLM can improve that workflow. It can also make the scanner more fluent while being wrong in more elaborate ways.\n\nThat is not progress. That is just a better-written interruption.\n\nThe interesting part of Mantis is not the label “agentic vulnerability scanning.” Everyone is attaching agentic to things now. Apparently software is not allowed to have a normal workflow anymore. The useful part is the structure around grounding: repository context, history, threat models, reviewer stages, critic stages, sandboxed reproduction, and patches that are tied back to evidence.\n\nThat shape makes sense because vulnerability detection is not one job. It is several jobs that often get collapsed into one vague “scan the code” box.\n\nThe reproduction step is the part I would care about most.\n\nIf a system can show a working crash, a failing test, an exploit path, or a concrete data-flow issue, the review conversation changes. The security team is no longer reading model confidence. They are reading evidence. That does not remove human judgment, but it gives the human something useful to judge.\n\nMost teams do not suffer because they have too few alerts. They suffer because the alerts arrive without enough context. Someone has to open the repository, understand the service boundary, trace input handling, inspect validation, check the framework behavior, look at previous fixes, decide whether the path is reachable, and then argue with the scanner’s output. That is expensive work.\n\nAI can help with that work, but only if the system is designed as a workflow rather than a magic box.\n\nFor example, a scanner that says:\n\nPossible SQL injection in UserController.java\n\nis not enough. A better system should be able to say:\n\n```\nThis route accepts user input here.\nThe value reaches this query builder here.\nThis sanitizer does not cover this pattern.\nThis is a minimal reproducer.\nThis is the failing test.\nThis is the proposed fix.\nThis is the residual uncertainty.\n```\n\nThat last line matters. Residual uncertainty is not weakness. It is honesty. A security tool that pretends every finding is equally certain creates bad incentives. Engineers start ignoring it, security teams start tuning it down, and eventually the tool becomes background noise.\n\nThe useful promise of an agentic scanner is that it can break the work into stages and let each stage challenge the previous one. One agent may identify suspicious flows, whereas another may criticize the finding, and another may try to reproduce it. Another may propose a patch. A final stage may check whether the patch changes behavior outside the intended area.\n\nThat is much closer to how a careful human review works.\n\nIt also changes how I would think about model selection. Not every stage needs the biggest model available. Classification, deduplication, clustering similar findings, and summarizing repository structure are not the same as tracing a subtle authorization bypass through five layers of application code. A practical system should spend reasoning budget where reasoning is actually needed.\n\nThat is a very SRE-ish way to think about AI, and I mean that as a compliment. The model is not magic. It is one component in a workflow with cost, latency, permissions, state, logs, and failure modes.\n\nThe cost piece is easy to ignore in a research announcement and painful to ignore in a real engineering organization. If every pull request triggers a deep multi-agent investigation across a large monorepo, the bill will get interesting very quickly. The system needs triage. It needs cheap filters, expensive analysis only when justified, and clear rules for what runs synchronously in CI versus what runs asynchronously in a security pipeline.\n\nThere is also a timing question. Some checks belong directly in the developer loop. They should run fast, fail clearly, and produce a result while the developer still remembers what they changed. Other checks are better as background analysis. A deep investigation across service boundaries may be valuable, but it probably should not block every commit unless the organization is prepared for that operational cost.\n\nThis is where the design should separate developer feedback from security investigation.\n\nDeveloper feedback needs speed and clarity. Security investigation needs depth and evidence. Trying to make one workflow satisfy both usually produces a system that is too slow for developers and too shallow for security teams. That is a familiar failure mode. We have seen it with static analysis, dependency scanning, data-quality checks, and policy-as-code.\n\nAI does not remove that trade-off. It just makes it easier to hide for a while.\n\nI would probably separate the workflow like this:\n\nThis keeps the expensive reasoning stages closer to the findings that deserve them.\n\nThere is still plenty of operational work hiding behind the nice diagram. The scanner needs sandboxing. It needs restricted network access. It needs deterministic handoffs between stages. It needs audit logs. It needs a way to prevent an aggressive false-positive filter from suppressing weak-looking findings that are actually real. It also needs ownership, because once a tool starts filing security bugs or proposing patches, someone has to decide what “good enough” means.\n\nThis is where security automation often becomes uncomfortable. A tool that only reports findings is easy to ignore. A tool that opens patches is harder to ignore, but also more dangerous. The patch might fix the immediate issue while changing behavior somewhere else. It might silence a test instead of fixing the cause. It might introduce a different vulnerability. It might be correct technically but wrong for the product’s authorization model.\n\nAuthorization bugs are a good example. They rarely live in one obvious line of code. The check may depend on route configuration, middleware behavior, tenant context, cached permissions, database filters, and assumptions in the UI. A scanner that sees only the controller method may miss the real boundary. A model with repository context may do better, but only if the system gives it the right evidence and then forces it to prove the path.\n\nThe same is true for deserialization, SSRF, file access, and dependency confusion. The interesting question is often not “does this function look suspicious?” It is “can untrusted input actually reach this dangerous capability under realistic conditions?” That requires reachability analysis, test construction, environment modeling, and sometimes domain knowledge about how the application is deployed.\n\nThis is where agents can be useful, but also where they can become overconfident. A fluent explanation of a possible exploit is not the same as exploitability. I would rather have a scanner that says “I cannot prove this yet” than one that files a confident but ungrounded critical finding.\n\nSo I would not give an AI scanner unlimited write access to a production codebase. I would start with evidence generation, then move to suggested patches behind review, then allow more automation only for narrow, well-tested classes of changes.\n\nSomething like:\n\nThe important part is that automation earns trust through measured behavior. Not vibes. Not a launch post. Measured behavior.\n\nThe metrics should reflect that. I would track more than “number of vulnerabilities found.” That metric is too easy to game. A useful program would track confirmed true positives, false-positive rate by category, time from finding to reproduction, time from reproduction to patch, developer review burden, escaped vulnerabilities, and whether suggested fixes survived regression testing.\n\nThose numbers tell you whether the system is improving the security process or merely producing activity.\n\nThere is also a governance concern. If the scanner learns from internal code, writes summaries to disk, executes reproducers in sandboxes, and uses multiple models, the organization needs to know where sensitive data goes. Security tooling often has broad repository access by design. That makes data handling, retention, model-provider boundaries, and access logs part of the architecture, not an appendix.\n\nI would treat an AI security scanner almost like a privileged internal service:\n\nThat is more work than running a command-line scanner. But if the system is going to reason over sensitive code and propose security patches, the extra discipline is not optional.\n\nThe bigger issue is that AI security tooling can easily become another alert generator. Teams do not need more findings. They need better evidence, better prioritization, and a shorter path from suspicion to validated fix.\n\nThat is where agentic workflows may actually help. Not because an agent can read a lot of files, although that helps. Not because it can write a patch, although that is useful. The real value is in connecting the steps that human reviewers already perform manually:\n\nThere is also a cultural side to this. Developers will trust the system faster if it explains itself in the language of the codebase. Security teams will trust it sooner if there is reproducible evidence. Engineering leaders will trust it faster if it reduces mean time to validated fix without flooding teams with noise. Those are different success metrics, and a serious platform needs to satisfy all three.\n\nMy current view is simple: `AI belongs in security scanning when it is treated as an evidence-generation system.`\n\nThe agent can suggest. The harness has to prove. That is enough for the first serious version.\n\nOnly after that, let us talk about autonomy.\n\n| References:", "url": "https://wpnews.pro/news/ai-security-scanning-needs-evidence-not-just-more-agents", "canonical_source": "https://dev.to/sudeephazra/ai-security-scanning-needs-evidence-not-just-more-agents-2mlb", "published_at": "2026-09-13 08:47:20+00:00", "updated_at": "2026-09-13 09:01:29.633844+00:00", "lang": "en", "topics": ["ai-agents", "ai-safety", "ai-tools", "developer-tools", "ai-infrastructure"], "entities": ["Google", "Mantis"], "alternates": {"html": "https://wpnews.pro/news/ai-security-scanning-needs-evidence-not-just-more-agents", "markdown": "https://wpnews.pro/news/ai-security-scanning-needs-evidence-not-just-more-agents.md", "text": "https://wpnews.pro/news/ai-security-scanning-needs-evidence-not-just-more-agents.txt", "jsonld": "https://wpnews.pro/news/ai-security-scanning-needs-evidence-not-just-more-agents.jsonld"}}