cd /news/artificial-intelligence/llm-agents-are-now-finding-zero-days… Β· home β€Ί topics β€Ί artificial-intelligence β€Ί article
[ARTICLE Β· art-14071] src=dev.to pub= topic=artificial-intelligence verified=true sentiment=↑ positive

LLM Agents Are Now Finding Zero-Days: How AI is Autonomously Rewriting the Rules of Vulnerability Research

LLM agents are now autonomously hunting zero-day vulnerabilities at massive scale, with Anthropic's Claude Mythos Preview finding over 10,000 critical or high-severity CVEs in under a month. In a landmark achievement, Apple credited Calif.io in collaboration with Claude and Anthropic Research for discovering CVE-2026-28952, a kernel-level privilege escalation vulnerability in macOS Tahoe 26.5 that allows arbitrary apps to gain root access. Unlike traditional scanners that match patterns, these AI agents reason about programmer intent versus actual behavior, chaining multiple low-severity bugs into high-severity exploit chains that human security teams had missed.

read19 min publishedMay 26, 2026

πŸ’‘

TL;DR:LLM agents like Claude Mythos Preview and GPT-5.5 are now autonomously hunting zero-days at massive scale β€” 10,000+ critical CVEs found in weeks. This post breaks down the agentic harness architecture, real-world results, and gives you runnable code to deploy your own AI security pipeline today.

Published: May 26, 2026 · ⏱️ 18 min read · Tags: security, llm, ai-agents, vulnerability-research, devops, cybersecurity

On May 11, 2026 β€” just days ago β€” Apple published its security advisory for macOS Tahoe 26.5. Tucked among dozens of credited human researchers was one unusual line:

CVE-2026-28952β€” An integer overflow addressed with improved input validation. Impact: An app may be able to gain root privileges.

Discovered by: Calif.io in collaboration with Claude and Anthropic Research.

Read that again. A kernel-level privilege escalation vulnerability β€” the kind that allows arbitrary apps to gain root access on macOS β€” was credited to an AI model.

This wasn't a toy benchmark or a controlled research sandbox. This was a real CVE, now patched and assigned by Apple, found in critical kernel code by a large language model operating as an autonomous security research agent. The same week, Anthropic's Project Glasswing announced that Claude Mythos Preview had found over 10,000 critical or high-severity vulnerabilities across the world's most systemically important software in under a month.

If you're a security engineer, a platform developer, or anyone who ships software that other people depend on β€” this changes your threat model. Permanently. This post breaks down exactly what happened, how these LLM vulnerability research agents work under the hood, and what you need to do about it right now.

Before LLMs, automated vulnerability detection fell into well-understood categories:

LLM vulnerability research is none of these β€” and all of them at once.

What makes frontier LLMs different is contextual reasoning at scale. A traditional SAST scanner matches patterns. An LLM understands what the code is trying to do, can reason about multi-file call graphs, can hypothesize about trust boundaries, and can generate the proof that a bug is exploitable β€” all in a single reasoning pass.

The key insight that the research community has arrived at in 2026 is this: LLMs don't just find bugs by recognizing patterns. They find bugs by understanding programmer intent vs. actual behavior β€” and finding where those diverge.

A 20-year-old XSLT bug in Firefox wasn't missed by fuzzers because the input space wasn't covered. It was missed because understanding the bug required knowing that reentrant key() calls cause a hash table rehash that frees its backing store while a raw entry pointer is still in use

β€” a multi-step logical chain that requires semantic understanding of the codebase's memory model. Claude Mythos found it.

This is the paradigm shift. We're no longer talking about automated scanners. We're talking about AI agents that reason like senior security researchers.

Cloudflare's security team spent weeks with Mythos Preview on their own infrastructure, and their writeup identified two capabilities that distinguish it from all prior tooling:

Real exploits rarely use a single vulnerability. They chain multiple primitives together β€” a use-after-free (UAF) becomes an arbitrary read/write primitive, which enables control-flow hijacking, which enables a full sandbox escape. Each step is individually low-severity; together they're critical.

Traditional scanners report bugs in isolation. Mythos Preview reasons about how to chain them. Given a set of identified primitives, it evaluates:

Cloudflare observed the model taking bugs that would normally sit ignored in a low-severity backlog and constructing high-severity exploit chains that their own security team hadn't considered. This isn't just vulnerability finding β€” it's vulnerability weaponization, in service of defenders understanding true risk.

Finding a bug and proving it's exploitable are two very different things. Mythos Preview closes this gap with an autonomous PoC generation loop:

This loop runs autonomously. Cloudflare described watching the model read compiler errors, adjust its exploit logic, and retry β€” behavior that previously required a human researcher sitting at a terminal. The result is a finding backed by a working proof of concept, not a speculative observation hedged with "might" and "potentially."

The numbers from Project Glasswing's first month are genuinely staggering:

Organization Bugs Found Severity Notes
Project Glasswing Partners (~50 orgs)
10,000+
Critical/High Collectively across critical infrastructure
Cloudflare
2,000
400 Critical/High Scanned 50+ internal repos
Mozilla Firefox
271
Mixed 10x more than Firefox 148 with Opus 4.6
Open Source Projects (1,000+)
6,202 (high/critical est.)
High/Critical 90.6% true-positive rate after triage
Palo Alto Networks
5x normal patch volume
β€” Accelerated release cadence

Mozilla's Hacks blog published their harness methodology and even disclosed specific bug IDs β€” an unusual level of transparency that gives us a rare window into what AI-found bugs actually look like in practice. A few highlights:

<legend>

HTML element triggered by an intricate orchestration of recursion stack depth limits, expando properties, and cycle collection across distant parts of the browser.These aren't simple buffer overflows. These are complex, multi-system, architecture-aware bugs that require deep understanding of browser internals. Fuzzers, which work by exploring input space, simply can't reason about the semantic relationships between components that make these bugs possible.

One important caveat: early LLM-based security scanning (2024–early 2025 era models) was plagued by AI-generated slop bug reports β€” plausible-sounding but entirely wrong findings that wasted maintainer time. Several open-source projects created policies explicitly rejecting AI-generated issues.

Mythos Preview represents a step-change improvement here. Cloudflare reported that the model's output had noticeably higher quality: fewer hedged findings, clearer reproduction steps, and less work to reach a fix-or-dismiss decision. Critically, findings backed by a working PoC have a false-positive rate that approaches zero by definition β€” if the exploit runs and produces the expected output, the bug is real.

The key lesson from all successful deployments is this: naΓ―vely pointing an LLM at a repository and asking "find bugs" doesn't work well. The quality of results scales dramatically with the sophistication of the harness around the model. Here's the architecture that state-of-the-art practitioners are converging on:

β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”
β”‚                    SECURITY AGENT PIPELINE                  β”‚
β”œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€
β”‚  1. THREAT MODELER          β”‚  Maps codebase, identifies    β”‚
β”‚     (LLM + static analysis) β”‚  attack surfaces, prioritizes β”‚
β”œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€
β”‚  2. SCANNER ORCHESTRATOR    β”‚  Spins up parallel sub-agents β”‚
β”‚     (Agent coordinator)     β”‚  per module/subsystem         β”‚
β”œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€
β”‚  3. VULN DETECTOR           β”‚  Per-file/function analysis   β”‚
β”‚     (LLM sub-agent)         β”‚  with semantic reasoning      β”‚
β”œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€
β”‚  4. EXPLOIT SYNTHESIZER     β”‚  Generates PoC code,          β”‚
β”‚     (LLM + code executor)   β”‚  compiles, and runs in sandboxβ”‚
β”œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€
β”‚  5. TRIAGE ENGINE           β”‚  Multi-model consensus,       β”‚
β”‚     (Ensemble of models)    β”‚  severity rating, dedup       β”‚
β”œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€
β”‚  6. REPORT GENERATOR        β”‚  CVE-formatted output,        β”‚
β”‚     (LLM)                   β”‚  fix suggestions, CVSS scoringβ”‚
β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜

The single biggest productivity multiplier is spending compute on threat modeling before scanning. Ask the LLM to:

This turns unfocused scanning into targeted analysis. Mozilla's team found this dramatically improved signal-to-noise: instead of 10,000 low-confidence findings across the whole codebase, they got 500 high-confidence findings in the highest-risk subsystems.

Each high-priority module gets its own sub-agent instance with:

async def scan_module(module_path: str, context: SecurityContext) -> list[Finding]:
    """
    Launch a sandboxed LLM sub-agent to analyze a single module.
    Returns structured findings with severity, description, and PoC.
    """
    system_prompt = build_security_analyst_prompt(
        language=context.language,
        vulnerability_classes=context.priority_vuln_classes,
        trust_model=context.trust_model,
        output_schema=FindingSchema
    )

    file_content = load_with_dependencies(module_path, context.repo_root)

    findings = await llm_client.chat(
        model="claude-opus-4-7",           # or gpt-5.5 for high-value targets
        system=system_prompt,
        messages=[{
            "role": "user",
            "content": f"Analyze this module for security vulnerabilities:\n\n{file_content}"
        }],
        response_schema=list[Finding],      # structured output enforces quality
        max_tokens=8192,
        timeout=120
    )

    return findings

This is where the magic happens β€” and where the false-positive rate collapses:

async def validate_finding(finding: Finding, sandbox: SandboxEnv) -> ValidatedFinding:
    """
    Attempt to generate and run a PoC for a suspected vulnerability.
    A finding backed by a working PoC has effectively 0% false positive rate.
    """
    max_iterations = 5

    for attempt in range(max_iterations):
        poc_code = await llm_client.chat(
            model="claude-opus-4-7",
            messages=[{
                "role": "user", 
                "content": f"""
                Write a minimal proof-of-concept that triggers this vulnerability:

                Finding: {finding.description}
                Affected code: {finding.code_snippet}
                Expected behavior: {finding.expected_trigger}

                Write executable {finding.language} code only. No explanations.
                """
            }]
        )

        result = await sandbox.execute(
            code=poc_code,
            language=finding.language,
            timeout=30,
            memory_limit="512mb"
        )

        if result.crashed and matches_expected_behavior(result, finding):
            return ValidatedFinding(
                finding=finding,
                poc_code=poc_code,
                execution_result=result,
                confidence="HIGH",
                false_positive=False
            )

        finding = await refine_hypothesis(finding, result, llm_client)

    return ValidatedFinding(finding=finding, confidence="LOW", false_positive=True)

One of the most powerful techniques for reducing false positives β€” borrowed from Milvus's research on AI code review β€” is running multiple independent models and requiring consensus. A finding reported by Claude Opus, GPT-5.5, and Gemini independently is orders of magnitude more likely to be real than one reported by a single model.

async def triage_with_consensus(
    finding: Finding,
    models: list[str] = ["claude-opus-4-7", "gpt-5.5", "gemini-2.5-pro"]
) -> ConsensusResult:
    """
    Submit a finding to multiple models for independent verification.
    Require 2/3 agreement to advance to human review queue.
    """
    verdicts = await asyncio.gather(*[
        verify_finding_with_model(finding, model) 
        for model in models
    ])

    confirmed_count = sum(1 for v in verdicts if v.is_valid)

    return ConsensusResult(
        finding=finding,
        verdicts=verdicts,
        consensus_reached=confirmed_count >= 2,
        confidence_score=confirmed_count / len(models),
        advance_to_human_review=confirmed_count >= 2
    )

As of May 2026, two models dominate the LLM vulnerability research space. Here's how they compare based on independent benchmarks and real-world deployments:

Capability Claude Mythos Preview GPT-5.5
Availability
Restricted (Project Glasswing / Enterprise) Generally available
Vulnerability Miss Rate
~5-8% (est.)
10% (XBOW benchmark)
Black-box performance
Excellent Excellent β€” outperforms GPT-5 with source code
White-box performance
Best-in-class "Effectively killed" XBOW's benchmark
Exploit chain construction
βœ… Core capability βœ… Strong
PoC generation
βœ… Autonomous loop βœ… Strong
Persist vs. pivot decision-making
Strong Improved (50% fewer bad persist decisions vs. GPT-5.4)
Consistency/guardrails
Inconsistent organic refusals More consistent behavior
Token efficiency
"Absolutely unprecedented precision" (XBOW) Good

The key practical difference today: Claude Mythos Preview is not publicly available β€” it's restricted to Project Glasswing partners and enterprise security teams with a verified use case. GPT-5.5 is generally available and, per XBOW's benchmarks, delivers Mythos-class performance in white-box scenarios.

For most security teams today, GPT-5.5 in a well-architected harness is the path to production. If your organization qualifies for Anthropic's Cyber Verification Program or Claude Security enterprise beta, Mythos-class capabilities are accessible via Claude Opus 4.7 as well.

Here's the uncomfortable truth that Project Glasswing has surfaced for the entire software industry:

AI has solved the hard part. The bottleneck is now entirely human.

For decades, the security community's limiting factor was finding vulnerabilities β€” it required expensive, senior human expertise and took weeks per codebase. That constraint has evaporated. Mythos Preview is finding critical bugs faster than any team of human researchers could. The new constraint is triage, disclosure, patch development, and deployment.

Some maintainers in Project Glasswing's open-source scanning initiative have asked Anthropic to slow down disclosure because they can't keep up. That's an extraordinary sentence. A world-class AI is producing so much valid, actionable security research that human maintainers are begging it to stop.

The downstream implications for your engineering organization:

Shorten patch cycles aggressively. The 90-day standard disclosure window was designed for the old world. As AI-found bugs become public CVEs faster, the exploitation window is compressing.

Invest in automated patch generation pipelines. Claude Security (now in public beta for Enterprise) can generate proposed fixes, not just identify bugs. This is the next frontier for reducing the triage burden.

Memory-safe languages matter more than ever. Both Cloudflare and Mozilla's data confirm significantly higher false-positive rates and more severe findings in C/C++ codebases vs. Rust or Go. The ROI on memory-safe rewrites just got a lot more concrete.

Staged rollout policies are critical. With AI accelerating both attack and defense, end users need to be able to receive patches faster. Frictionless update mechanisms aren't just a UX concern β€” they're a security posture.

You don't need access to Mythos Preview to start today. Here's a practical, production-ready approach using generally available models:

#!/usr/bin/env python3
"""
minimal_vuln_scanner.py
A basic LLM-powered vulnerability scanner for CI/CD integration.
Requires: anthropic>=0.30.0, pip install anthropic
"""

import asyncio
import json
from pathlib import Path
from anthropic import AsyncAnthropic

client = AsyncAnthropic()

SECURITY_SYSTEM_PROMPT = """You are an expert security researcher performing a 
white-box vulnerability audit. Analyze the provided code for:

1. Memory safety issues (buffer overflows, UAF, null deref β€” especially in C/C++)
2. Injection vulnerabilities (SQL, command, LDAP, path traversal)  
3. Authentication/authorization bypasses
4. Race conditions and TOCTOU bugs
5. Cryptographic weaknesses
6. Unsafe deserialization
7. Integer overflow/underflow conditions
8. Logic bugs affecting security-critical code paths

For each finding, provide:
- Vulnerability class (CWE ID if applicable)
- Severity (Critical/High/Medium/Low)
- Affected code location (file:line)
- Root cause explanation (2-3 sentences)
- Proof-of-concept trigger (how would an attacker trigger this?)
- Recommended fix

Return your response as a JSON array of findings. If no vulnerabilities are found,
return an empty array []. Do NOT speculate β€” only report findings you are confident about."""

async def scan_file(file_path: Path) -> list[dict]:
    """Scan a single file for vulnerabilities using Claude."""

    content = file_path.read_text(errors='replace')

    if len(content.strip()) < 50:
        return []

    message = await client.messages.create(
        model="claude-opus-4-5",  # Use claude-opus-4-7 for higher accuracy
        max_tokens=4096,
        system=SECURITY_SYSTEM_PROMPT,
        messages=[{
            "role": "user",
            "content": f"File: {file_path}\n\n```
{% endraw %}
\n{content[:50000]}\n
{% raw %}
```"
        }]
    )

    response_text = message.content[0].text.strip()

    try:
        start = response_text.find('[')
        end = response_text.rfind(']') + 1
        if start != -1 and end > start:
            findings = json.loads(response_text[start:end])
            for f in findings:
                f['source_file'] = str(file_path)
            return findings
    except json.JSONDecodeError:
        pass

    return []

async def scan_repository(repo_path: str, extensions: list[str] = None) -> dict:
    """
    Scan an entire repository for vulnerabilities.

    Args:
        repo_path: Path to the repository root
        extensions: File extensions to scan (default: common security-relevant types)

    Returns:
        Dict with findings grouped by severity
    """
    if extensions is None:
        extensions = ['.c', '.cpp', '.h', '.py', '.js', '.ts', '.go', '.rs', '.java']

    repo = Path(repo_path)
    files_to_scan = [
        f for f in repo.rglob('*')
        if f.suffix in extensions
        and '.git' not in f.parts
        and 'node_modules' not in f.parts
        and 'vendor' not in f.parts
    ]

    print(f"[*] Scanning {len(files_to_scan)} files in {repo_path}")

    semaphore = asyncio.Semaphore(5)  # Max 5 concurrent API calls

    async def scan_with_limit(f):
        async with semaphore:
            print(f"    Scanning: {f.relative_to(repo)}")
            return await scan_file(f)

    all_results = await asyncio.gather(*[scan_with_limit(f) for f in files_to_scan])

    all_findings = [f for sublist in all_results for f in sublist]

    grouped = {
        'critical': [f for f in all_findings if f.get('severity', '').lower() == 'critical'],
        'high':     [f for f in all_findings if f.get('severity', '').lower() == 'high'],
        'medium':   [f for f in all_findings if f.get('severity', '').lower() == 'medium'],
        'low':      [f for f in all_findings if f.get('severity', '').lower() == 'low'],
    }

    return grouped

async def main():
    import sys
    repo_path = sys.argv[1] if len(sys.argv) > 1 else '.'

    results = await scan_repository(repo_path)

    total = sum(len(v) for v in results.values())
    print(f"\n{'='*60}")
    print(f"SCAN COMPLETE β€” {total} findings")
    print(f"{'='*60}")
    print(f"  πŸ”΄ Critical: {len(results['critical'])}")
    print(f"  🟠 High:     {len(results['high'])}")
    print(f"  🟑 Medium:   {len(results['medium'])}")
    print(f"  🟒 Low:      {len(results['low'])}")
    print(f"{'='*60}\n")

    for severity in ['critical', 'high']:
        for finding in results[severity]:
            print(f"[{finding['severity'].upper()}] {finding.get('vulnerability_class', 'Unknown')}")
            print(f"  File: {finding.get('source_file')}")
            print(f"  {finding.get('root_cause', 'No description')}")
            print(f"  Fix: {finding.get('recommended_fix', 'See full report')}\n")

    with open('security_report.json', 'w') as f:
        json.dump(results, f, indent=2)
    print("[*] Full report saved to security_report.json")

if __name__ == '__main__':
    asyncio.run(main())
name: AI Security Scan

on:
  pull_request:
    types: [opened, synchronize]
  schedule:
    - cron: '0 2 * * 1'  # Weekly full scan every Monday at 2am

jobs:
  llm-vuln-scan:
    runs-on: ubuntu-latest
    permissions:
      pull-requests: write
      security-events: write

    steps:
      - uses: actions/checkout@v4
        with:
          fetch-depth: 0  # Full history for diff-based scanning on PRs

      - name: Set up Python
        uses: actions/setup-python@v5
        with:
          python-version: '3.12'

      - name: Install dependencies
        run: pip install anthropic>=0.30.0

      - name: Run AI Security Scanner
        env:
          ANTHROPIC_API_KEY: ${{ secrets.ANTHROPIC_API_KEY }}
        run: |
          if [ "${{ github.event_name }}" = "pull_request" ]; then
            git diff --name-only origin/${{ github.base_ref }}...HEAD > changed_files.txt
            python minimal_vuln_scanner.py --files-list changed_files.txt
          else
            python minimal_vuln_scanner.py .
          fi

      - name: Check for Critical Findings
        run: |
          CRITICAL_COUNT=$(python -c "
          import json
          with open('security_report.json') as f:
              report = json.load(f)
          print(len(report.get('critical', [])))
          ")
          echo "Critical findings: $CRITICAL_COUNT"
          if [ "$CRITICAL_COUNT" -gt "0" ]; then
            echo "::error::$CRITICAL_COUNT critical security vulnerabilities found!"
            exit 1
          fi

      - name: Post PR Comment with Findings
        if: github.event_name == 'pull_request'
        uses: actions/github-script@v7
        with:
          script: |
            const fs = require('fs');
            const report = JSON.parse(fs.readFileSync('security_report.json'));
            const total = Object.values(report).flat().length;

            const body = `## πŸ” AI Security Scan Results

            | Severity | Count |
            |---|---|
            | πŸ”΄ Critical | ${report.critical?.length || 0} |
            | 🟠 High | ${report.high?.length || 0} |
            | 🟑 Medium | ${report.medium?.length || 0} |
            | 🟒 Low | ${report.low?.length || 0} |

            ${total === 0 ? 'βœ… No vulnerabilities found!' : '⚠️ Review findings in the security_report.json artifact.'}`;

            github.rest.issues.createComment({
              issue_number: context.issue.number,
              owner: context.repo.owner,
              repo: context.repo.repo,
              body: body
            });

For high-value codebases, the production-grade approach is multi-model consensus to approach near-zero false-positive rates:


import asyncio
import json
from anthropic import AsyncAnthropic
from openai import AsyncOpenAI

anthropic_client = AsyncAnthropic()
openai_client = AsyncOpenAI()

VERIFICATION_PROMPT = """You are an expert security researcher verifying whether
a reported vulnerability is real or a false positive.

Given the following finding and source code, answer:
1. Is this vulnerability real? (yes/no/uncertain)
2. If real: can an attacker trigger it from an untrusted context? (yes/no/uncertain)
3. Confidence: (high/medium/low)

Respond in JSON: {"is_real": bool, "triggerable": bool, "confidence": "high"|"medium"|"low", "reasoning": "one sentence"}"""

async def verify_with_claude(finding: dict, source_code: str) -> dict:
    msg = await anthropic_client.messages.create(
        model="claude-opus-4-5",
        max_tokens=512,
        system=VERIFICATION_PROMPT,
        messages=[{"role": "user", "content": f"Finding:\n{json.dumps(finding)}\n\nCode:\n{source_code}"}]
    )
    return json.loads(msg.content[0].text)

async def verify_with_gpt(finding: dict, source_code: str) -> dict:
    resp = await openai_client.chat.completions.create(
        model="gpt-4.1",
        messages=[
            {"role": "system", "content": VERIFICATION_PROMPT},
            {"role": "user", "content": f"Finding:\n{json.dumps(finding)}\n\nCode:\n{source_code}"}
        ],
        max_tokens=512,
        response_format={"type": "json_object"}
    )
    return json.loads(resp.choices[0].message.content)

async def consensus_verify(finding: dict, source_code: str) -> dict:
    """Verify a finding with multiple models; return consensus result."""
    claude_result, gpt_result = await asyncio.gather(
        verify_with_claude(finding, source_code),
        verify_with_gpt(finding, source_code)
    )

    both_confirm = claude_result.get('is_real') and gpt_result.get('is_real')

    return {
        "finding": finding,
        "consensus": both_confirm,
        "claude_verdict": claude_result,
        "gpt_verdict": gpt_result,
        "advance_to_human_review": both_confirm,
        "false_positive_probability": "low" if both_confirm else "high"
    }

It would be irresponsible to discuss this technology without addressing the elephant in the room: the same capability that finds bugs for defenders can be used by attackers.

Anthropic has been explicit about this tension. From their Glasswing update:

"Models as capable as Mythos Preview will soon be developed by many different AI companies. At present, no company β€” including Anthropic β€” has developed safeguards strong enough to prevent such models from being misused."

This is why Mythos Preview is not publicly released. But it's also why this matters: the capability genie is not going back in the bottle. The question isn't whether powerful AI vulnerability research tools will exist β€” they will. The question is whether defenders can gain and hold an asymmetric advantage before those tools proliferate to malicious actors.

Key ethical considerations for engineers building in this space:

Responsible disclosure, always. AI is going to accelerate vulnerability discovery dramatically. The 90-day disclosure standard exists for good reason β€” it gives end users time to patch. Don't let the excitement of AI-found bugs shortcut this process.

Scope your harness carefully. Ensure your scanning pipeline only touches infrastructure you own or have explicit written authorization to test. The fact that a tool is effective doesn't change the legal and ethical requirements for authorization.

Verify before you disclose. Submit only confirmed, PoC-backed findings to maintainers. The open-source community is already overwhelmed by low-quality AI-generated bug reports. Be part of the solution, not the problem.

Watch for model inconsistency. Cloudflare's team documented that Mythos Preview's organic guardrails are inconsistent β€” the same task framed differently could produce completely different refusal behavior. Don't treat model-level safeguards as a substitute for process-level controls.

Based on the current trajectory, here's what the next 12–18 months look like for LLM vulnerability research:

Near-term (3–6 months):

Medium-term (6–18 months):

Long-term:

We are living through a genuine phase transition in software security. The tools that found a kernel CVE in macOS, 271 latent bugs in Firefox, and 2,000 vulnerabilities across Cloudflare's infrastructure in weeks β€” these are not research prototypes. They are production systems, available today, finding real bugs in real code.

The LLM vulnerability research agent isn't coming. It's here. And if you're shipping software that other people depend on, the question is not whether to engage with this technology β€” it's whether you engage with it proactively, as a defender, or reactively, after an attacker already has.

Three things you can do this week:

Run the minimal scanner above against your most critical service. Set your ANTHROPIC_API_KEY, point it at a repo, and see what it finds. The marginal cost of a scan is a few API dollars. The marginal cost of an unpatched critical is not.

Set up the GitHub Actions workflow for your team's most security-sensitive repositories. Automated scanning on every PR is now table stakes.

Apply to Anthropic's Cyber Verification Program if your organization does legitimate security research, red-teaming, or penetration testing. Access to higher-capability models in this domain is now a significant professional advantage.

The Glasswing era of software security has begun. The organizations that understand the architecture behind these tools β€” not just that they exist, but how they work and how to deploy them effectively β€” will have a structural security advantage for the next decade.

The bugs are being found. The question is who finds them first.

Found this useful? Drop a ⭐ on the companion GitHub repo with the full harness implementation, contribute to the discussion in the comments, and share this with the security engineer on your team who hasn't heard about Project Glasswing yet.

Tags: llm-vulnerability-research

generative-ai

cybersecurity

agentic-ai

claude

gpt-5

devsecops

security-engineering

zero-day

project-glasswing

── more in #artificial-intelligence 4 stories Β· sorted by recency
sponsored brought to you by zahid.host 4,200+ EU-deployed projects
reading about agents? ship yours in a single git push.

Run your AI side-project on zahid.host

EU-based hosting, git-push deploys, automatic HTTPS, no cold starts. Free tier with a custom domain β€” perfect for shipping the agent you just read about.

$git push zahid main
β†’ Live at https://your-agent.zahid.host βœ“
Get free account β†’ Pricing
from €0/mo Β· no card required
LIVE [news/llm-agents-are-now-f…] indexed:0 read:19min 2026-05-26 Β· β€”