My AI pentest agent reported 23 root shells. It had actually popped zero.

A developer building an autonomous penetration-testing agent found that an LLM-driven system falsely reported 23 root shells when it had actually popped zero. The agent's heuristic for confirming breaches relied on string matches like "Shellcodes" in searchsploit output, leading to inflated success claims. By replacing the heuristic with evidence-based checks for actual shell markers, the agent now reports only three real root shells and honestly acknowledges failures.

I've been building an autonomous penetration-testing agent — an LLM driving real tools nmap, masscan, hydra, Metasploit, searchsploit around a loop: recon a target, pick an exploit, fire it, decide whether it worked, move on. Everything below runs against a deliberately vulnerable Metasploitable VM in an isolated lab. Nothing here is a technique for attacking systems you don't own; it's a story about making an AI agent tell the truth about what it did. Because the first hard lesson was this: an LLM agent will happily report success it never achieved. One of my early runs produced a beautiful report — This post is the arc from that lie to an engine that now pops three real root shells, reports them honestly, and refuses to claim anything it can't prove. The validator — the component that decides whether an exploit worked — was doing something that looks reasonable and is catastrophic: python the original "did it work?" check if "login:" in output or "shellcodes" in output.lower : return "confirmed" Two independent failure modes fed this: searchsploit output. When you search Exploit-DB, the tool prints a header: Exploits: ... / Shellcodes: .... Every single search — pure intel, no exploitation whatsoever — contained the word "Shellcodes". So every port the agent looked at got marked breached. Banner text. Any service that echoed login: counted as a shell. The result was a report that was 100% adjectives and 0% evidence. "Confirmed." "High confidence." "Breached." All string matches, none of them a shell. The fix: evidence, not vibes I replaced the heuristic with breach confirmed — a function that only returns true when the output contains actual proof of code execution: SHELL EVIDENCE = re.compile r"uid=\d+\ a-z +\ . gid=\d+" , id 1 output re.compile r"root@ \w.- +: ~/ " , a root prompt ...markers a real shell produces, not ones a banner can fake def breach confirmed output: str - bool: return any p.search output for p in SHELL EVIDENCE A curated exploit that pops vsftpd 2.3.4's backdoor and runs id produces uid=0 root gid=0 root . That is a breach. A searchsploit table is not. The same run that used to claim 23/23 now says: 3 confirmed, 20 correctly unconfirmed. The 3 are real. That honesty — being willing to say "I tried and it didn't work" — is the single most important property of the whole system. Why it couldn't pop anything: the boring plumbing bugs Once the agent stopped lying, it exposed how little was actually landing. The culprits weren't clever — they were the unglamorous, silent kind that never throw a stack trace where you're looking: 1. The tool the model could never call. The model kept calling searchsploit with {"query": "vsftpd"}. The tool's schema required {"keyword": "..."}. The MCP layer hard-rejected every call as malformed before it ran — 20 times a run, each a silent 0-second non-event. Fix: accept keyword | query | search and drop the rigid required. 2. ANSI codes poisoning module names. msfconsole highlights your search term with color escape codes: exploit/unix/\x1b 45mftp\x1b 0m/vsftpd 234. My parser dutifully captured those bytes as part of the module path, so every selected module failed to load. Gated Metasploit was at a 0% success rate for a reason that was invisible in plain-text logs. Fix: strip ansi before parsing. 3. Version strings masquerading as products. nmap reports RPC services version-first: 2 RPC 100000 . My fingerprint parser grabbed 2 as the product name and fed "2" to Metasploit as a search term. Fix: a leading-digit guard so a version can never be mistaken for a product. 4. A 5-minute hang from one timeout constant. call model reused the 300-second tool timeout, so a single wedged local-LLM request stalled the entire engagement for five minutes. Fix: a separate 90-second model timeout. 5. A sandbox that ate its own output. The code-exec sandbox raised TimeoutExpired without printing the EXIT contract the parser expected, so a slow exploit surfaced as an opaque "No EXIT marker" after burning the whole wall clock. Fix: catch the timeout, emit a clean EXIT 124 with partial output, drop the default wall to 60s. None of these are interesting. All of them were the difference between "works" and "silently does nothing." Collectively they were kneecapping the agent while the logs looked fine. Why it fired garbage: relevance vs. relevance With the plumbing fixed, the agent started reaching Metasploit — and immediately did something absurd. Watch what it picked: Port Service fingerprint Module it chose Rank 23 Linux telnetd exploit/linux/http/asuswrt lan rce excellent 513 rlogin login exploit/windows/misc/ais esel server rce excellent 5900 VNC exploit/linux/misc/igel command injection excellent A router RCE against telnet. A Windows exploit against a Linux rlogin service. Every one "excellent"-ranked, every one nonsense. The reason: msfconsole search <term matches your term against module descriptions, not just their targets. A fuzzy one-word fingerprint like "login" pulls in any module whose blurb mentions logging in — and Metasploit ranks by exploit reliability, not by relevance to your target. So the best-ranked match to a bad query is confidently, precisely wrong. The relevance gate The fix is a filter that runs before ranking: keep a candidate only if a real service token from the fingerprint appears in the module's path — after throwing away the generic tree words that match everything. GENERIC = {"linux", "windows", "unix", "multi", "http", "misc", "scanner", "auxiliary", "exploit", ...} def relevance tokens terms: str - set: return {w for w in terms.lower .split if len w = 3 and not w 0 .isdigit and w not in GENERIC} def is relevant module: str, tokens: set - bool: return any tok in module for tok in tokens token must be in the PATH asuswrt, ais esel, igel — all gone. x11 keyboard exec for X11 and vnc keyboard exec for VNC survive. It trades a little recall for a lot of precision, which is the right trade when the failure mode is firing exploits at the wrong target. A quieter bug the gate revealed While I was in there: r-services modules rsh/rlogin/rexec live under auxiliary/scanner/rservices/, not exploit/. My selector had an exploits only=True flag that silently discarded the entire auxiliary tree — so the right module for rlogin could never be chosen, on any target. I removed the flag and switched to a tiered sort: exploit/ modules first they pop shells , auxiliary logins as a ranked fallback. No per-port hardcoding — it generalizes to any service Metasploit has coverage for. Going multi-agent — without re-importing the lie The next phase was turning a single-agent loop into a proper pipeline: orchestrator → recon → attacker → validator → reporter. The trap: those six agents already existed as disconnected demos, and the old validator used the exact same string-heuristic that produced "23/23." Wiring them in as-was would have re-imported the original sin. So the rule became: one honest engine, shared by everyone. I extracted the proven logic — AgentMemory, plan exploit step, breach confirmed — into a single exploitation core.py. The multi-agent validator now delegates to breach confirmed instead of matching strings. The attacker executes only through an injected, human-gated execute fn — never the ungated MCP client that bypasses the operator approval gate. That invariant is pinned by a test that fails if the bypass path is even touched: def test attacker never uses ungated path monkeypatch : monkeypatch.setattr mcp client, "call tool", boom raises if called asyncio.run run attacker gated ..., execute fn=fake gate green only if every exploit went through the gate Human-in-the-loop stays non-negotiable: exploitation is two-phase operator-approved, and the scope gate refuses anything outside the authorized target. The agent is autonomous about choosing; it is not autonomous about firing. The one-character bug that ate an afternoon The last one is my favorite, because it's so dumb and it hid so well. The orchestrated pipeline kept getting refused at startup: 🎯 engage multi 10.0.0.5 🚫 multi 10.0.0.5 refused at engagement start The command was engage multi <target a space . The dispatcher only matched engage-multi a hyphen . So engage multi 10.0.0.5 fell through to the single-agent engage branch, which stripped "engage " and left the target as the literal string "multi 10.0.0.5" — which the scope gate, entirely correctly, refused. The word "multi" was riding inside the target the whole time, right there in the log, and I read past it a dozen times. The fix is a boring pure function that accepts both separators, and eight tests so it never regresses: def parse engagement command goal: str : s, low = goal.strip , goal.strip .lower for prefix in "engage-multi ", "engage multi " : if low.startswith prefix : return "multi", s len prefix : .strip if low.startswith "engage " : return "single", s len "engage " : .strip return None, s With that, engage multi finally entered the orchestrated engine end-to-end: recon → attack → validate → report, three real root shells vsftpd, ingreslock, UnrealIRCd — all uid=0 , zero fakes, and the relevance gate holding X11→x11, VNC→vnc, no router RCEs at telnet . What I'd tell anyone building an agent that does things Measure success by artifacts, not adjectives. "Confirmed" is a word an LLM loves to emit. uid=0 root is a fact. Build your success check out of facts, and make it hard to satisfy. Your agent will report the outcome you hoped for unless evidence forbids it. The false-positive isn't a model quirk you prompt away; it's a system property you engineer against. The boring bugs cost the most. A wrong param name, an ANSI escape code, a shared timeout constant — none throw errors where you're looking, and together they can render a "working" agent completely inert. Keep the human on the trigger. Autonomy in target selection is useful. Autonomy in pulling the trigger on a real exploit is a liability. Gate it, log every decision, and let a test fail if anything routes around the gate. TDD is what let me refactor a lying engine into an honest one without losing the parts that already worked — 240+ tests, red-green on every change, and the false-positive class permanently fenced off by regression tests. The agent still isn't "done" — the orchestrated attacker is a hair less thorough than the single-agent loop on a couple of services, and there's module option-tuning left before r-services actually pops. But it no longer lies to me, and after "23 of 23," that's the feature I care about most. Built and tested exclusively against an isolated, self-owned Metasploitable lab. If you build something like this, keep it in a lab you own too. --- 2 — Hacker News version short, link-first Format: HN wants a title + a url + optional short text. Use the title line as the submission title, put the GitHub or article link in the URL field, and paste the body as the first comment — HN readers respond well to an author's short "here's the story" comment. Title: Show HN: I made my AI pentest agent stop lying about what it hacked URL: https://github.com/XenoCoreGiger31/GEMMA-by-GOOGLE First-comment text: I've been building an autonomous pentest agent — an LLM driving real tools nmap, hydra, Metasploit, searchsploit around a recon → exploit → verify loop, against a Metasploitable VM in an isolated lab. The first hard lesson had nothing to do with exploitation and everything to do with honesty: one early run produced a confident report claiming "23 of 23 ports breached, root on all." The real number was zero. Not one shell. The cause was that "success" was a string match. The validator marked a port breached if the output contained login: or shellcodes — and searchsploit prints "Shellcodes:" in every search header, so merely looking at a port flagged it as owned. I replaced it with an evidence check that only passes on real proof of code execution uid=0 root , a root prompt . The same run then honestly reported 3 confirmed, 20 correctly unconfirmed. The 3 were real. Then the honest engine exposed how little was landing, for deeply boring reasons: a required param name the model never sent every searchsploit call hard-rejected before running ; ANSI color codes from msfconsole poisoning every parsed module path 100% "failed to load" ; an nmap version string parsed as a product name; a shared 300s timeout letting one hung LLM call stall for five minutes. And a relevance problem: msfconsole search matches module descriptions, so a fuzzy one-word fingerprint fired an "excellent"-ranked asuswrt router RCE at a telnet port, and a Windows exploit at a Linux rlogin service. The fix keeps a candidate only if a real service token appears in the module path. Full write-up with the code for each fix is in the repo. Built and tested only against a self-owned lab; exploitation stays human-gated. Happy to talk about the false-positive class — I think it generalizes to any agent that reports on its own actions.