{"slug": "my-ai-pentest-agent-reported-23-root-shells-it-had-actually-popped-zero", "title": "My AI pentest agent reported 23 root shells. It had actually popped zero.", "summary": "A developer building an autonomous penetration-testing agent found that an LLM-driven system falsely reported 23 root shells when it had actually popped zero. The agent's heuristic for confirming breaches relied on string matches like \"Shellcodes\" in searchsploit output, leading to inflated success claims. By replacing the heuristic with evidence-based checks for actual shell markers, the agent now reports only three real root shells and honestly acknowledges failures.", "body_md": "I've been building an autonomous penetration-testing agent — an LLM driving real\n\ntools (nmap, masscan, hydra, Metasploit, searchsploit) around a loop: recon a\n\ntarget, pick an exploit, fire it, decide whether it worked, move on. Everything\n\nbelow runs against a deliberately vulnerable **Metasploitable** VM in an isolated\n\nlab. Nothing here is a technique for attacking systems you don't own; it's a\n\nstory about making an AI agent *tell the truth* about what it did.\n\nBecause the first hard lesson was this: **an LLM agent will happily report\nsuccess it never achieved.** One of my early runs produced a beautiful report —\n\nThis post is the arc from that lie to an engine that now pops three real root\n\nshells, reports them honestly, and refuses to claim anything it can't prove.\n\nThe validator — the component that decides whether an exploit worked — was doing\n\nsomething that looks reasonable and is catastrophic:\n\n```\npython\n# the original \"did it work?\" check\nif \"login:\" in output or \"shellcodes\" in output.lower():\n    return \"confirmed\"\nTwo independent failure modes fed this:\n\nsearchsploit output. When you search Exploit-DB, the tool prints a\nheader: Exploits: ... / Shellcodes: .... Every single search — pure intel,\nno exploitation whatsoever — contained the word \"Shellcodes\". So every port\nthe agent looked at got marked breached.\nBanner text. Any service that echoed login: counted as a shell.\nThe result was a report that was 100% adjectives and 0% evidence. \"Confirmed.\"\n\"High confidence.\" \"Breached.\" All string matches, none of them a shell.\n\nThe fix: evidence, not vibes\nI replaced the heuristic with breach_confirmed() — a function that only returns\ntrue when the output contains actual proof of code execution:\n\n_SHELL_EVIDENCE = (\n    re.compile(r\"uid=\\d+\\([a-z]+\\).*gid=\\d+\"),   # id(1) output\n    re.compile(r\"root@[\\w.-]+:[~/]\"),            # a root prompt\n    # ...markers a real shell produces, not ones a banner can fake\n)\n\ndef breach_confirmed(output: str) -> bool:\n    return any(p.search(output) for p in _SHELL_EVIDENCE)\nA curated exploit that pops vsftpd 2.3.4's backdoor and runs id produces\nuid=0(root) gid=0(root). That is a breach. A searchsploit table is not. The\nsame run that used to claim 23/23 now says: 3 confirmed, 20 correctly\nunconfirmed. The 3 are real. That honesty — being willing to say \"I tried and\nit didn't work\" — is the single most important property of the whole system.\n\nWhy it couldn't pop anything: the boring plumbing bugs\nOnce the agent stopped lying, it exposed how little was actually landing. The\nculprits weren't clever — they were the unglamorous, silent kind that never throw\na stack trace where you're looking:\n\n1. The tool the model could never call. The model kept calling\nsearchsploit with {\"query\": \"vsftpd\"}. The tool's schema required\n{\"keyword\": \"...\"}. The MCP layer hard-rejected every call as malformed\nbefore it ran — 20 times a run, each a silent 0-second non-event. Fix: accept\nkeyword | query | search and drop the rigid required.\n\n2. ANSI codes poisoning module names. msfconsole highlights your search\nterm with color escape codes: exploit/unix/\\x1b[45mftp\\x1b[0m/vsftpd_234. My\nparser dutifully captured those bytes as part of the module path, so every\nselected module failed to load. Gated Metasploit was at a 0% success rate for a\nreason that was invisible in plain-text logs. Fix: strip_ansi() before\nparsing.\n\n3. Version strings masquerading as products. nmap reports RPC services\nversion-first: 2 (RPC #100000). My fingerprint parser grabbed 2 as the\nproduct name and fed \"2\" to Metasploit as a search term. Fix: a leading-digit\nguard so a version can never be mistaken for a product.\n\n4. A 5-minute hang from one timeout constant. call_model() reused the\n300-second tool timeout, so a single wedged local-LLM request stalled the\nentire engagement for five minutes. Fix: a separate 90-second model timeout.\n\n5. A sandbox that ate its own output. The code-exec sandbox raised\nTimeoutExpired without printing the EXIT contract the parser expected, so a\nslow exploit surfaced as an opaque \"No EXIT marker\" after burning the whole wall\nclock. Fix: catch the timeout, emit a clean EXIT 124 with partial output, drop\nthe default wall to 60s.\n\nNone of these are interesting. All of them were the difference between \"works\"\nand \"silently does nothing.\" Collectively they were kneecapping the agent while\nthe logs looked fine.\n\nWhy it fired garbage: relevance vs. relevance\nWith the plumbing fixed, the agent started reaching Metasploit — and immediately\ndid something absurd. Watch what it picked:\n\nPort    Service (fingerprint)   Module it chose Rank\n23  Linux telnetd   exploit/linux/http/asuswrt_lan_rce  excellent\n513 rlogin (login)  exploit/windows/misc/ais_esel_server_rce    excellent\n5900    VNC exploit/linux/misc/igel_command_injection   excellent\nA router RCE against telnet. A Windows exploit against a Linux rlogin\nservice. Every one \"excellent\"-ranked, every one nonsense.\n\nThe reason: msfconsole search <term> matches your term against module\ndescriptions, not just their targets. A fuzzy one-word fingerprint like\n\"login\" pulls in any module whose blurb mentions logging in — and Metasploit\nranks by exploit reliability, not by relevance to your target. So the\nbest-ranked match to a bad query is confidently, precisely wrong.\n\nThe relevance gate\nThe fix is a filter that runs before ranking: keep a candidate only if a real\nservice token from the fingerprint appears in the module's path — after\nthrowing away the generic tree words that match everything.\n\n_GENERIC = {\"linux\", \"windows\", \"unix\", \"multi\", \"http\", \"misc\",\n            \"scanner\", \"auxiliary\", \"exploit\", ...}\n\ndef _relevance_tokens(terms: str) -> set:\n    return {w for w in terms.lower().split()\n            if len(w) >= 3 and not w[0].isdigit() and w not in _GENERIC}\n\ndef _is_relevant(module: str, tokens: set) -> bool:\n    return any(tok in module for tok in tokens)   # token must be in the PATH\nasuswrt, ais_esel, igel — all gone. x11_keyboard_exec for X11 and\nvnc_keyboard_exec for VNC survive. It trades a little recall for a lot of\nprecision, which is the right trade when the failure mode is firing exploits at\nthe wrong target.\n\nA quieter bug the gate revealed\nWhile I was in there: r-services modules (rsh/rlogin/rexec) live under\nauxiliary/scanner/rservices/, not exploit/. My selector had an\nexploits_only=True flag that silently discarded the entire auxiliary tree — so\nthe right module for rlogin could never be chosen, on any target. I removed the\nflag and switched to a tiered sort: exploit/ modules first (they pop shells),\nauxiliary logins as a ranked fallback. No per-port hardcoding — it generalizes to\nany service Metasploit has coverage for.\n\nGoing multi-agent — without re-importing the lie\nThe next phase was turning a single-agent loop into a proper pipeline:\norchestrator → recon → attacker → validator → reporter. The trap: those six\nagents already existed as disconnected demos, and the old validator used the\nexact same string-heuristic that produced \"23/23.\" Wiring them in as-was would\nhave re-imported the original sin.\n\nSo the rule became: one honest engine, shared by everyone. I extracted the\nproven logic — AgentMemory, plan_exploit_step, breach_confirmed — into a\nsingle exploitation_core.py. The multi-agent validator now delegates to\nbreach_confirmed instead of matching strings. The attacker executes only\nthrough an injected, human-gated execute_fn — never the ungated MCP client that\nbypasses the operator approval gate. That invariant is pinned by a test that\nfails if the bypass path is even touched:\n\ndef test_attacker_never_uses_ungated_path(monkeypatch):\n    monkeypatch.setattr(mcp_client, \"call_tool\",\n                        _boom)  # raises if called\n    asyncio.run(run_attacker_gated(..., execute_fn=fake_gate))\n    # green only if every exploit went through the gate\nHuman-in-the-loop stays non-negotiable: exploitation is two-phase\noperator-approved, and the scope gate refuses anything outside the authorized\ntarget. The agent is autonomous about choosing; it is not autonomous about\nfiring.\n\nThe one-character bug that ate an afternoon\nThe last one is my favorite, because it's so dumb and it hid so well. The\norchestrated pipeline kept getting refused at startup:\n\n🎯 engage multi 10.0.0.5\n🚫 multi 10.0.0.5 refused at engagement start\nThe command was engage multi <target> (a space). The dispatcher only matched\nengage-multi (a hyphen). So engage multi 10.0.0.5 fell through to the\nsingle-agent engage branch, which stripped \"engage \" and left the target\nas the literal string \"multi 10.0.0.5\" — which the scope gate, entirely\ncorrectly, refused. The word \"multi\" was riding inside the target the whole time,\nright there in the log, and I read past it a dozen times.\n\nThe fix is a boring pure function that accepts both separators, and eight tests\nso it never regresses:\n\ndef parse_engagement_command(goal: str):\n    s, low = goal.strip(), goal.strip().lower()\n    for prefix in (\"engage-multi \", \"engage multi \"):\n        if low.startswith(prefix):\n            return \"multi\", s[len(prefix):].strip()\n    if low.startswith(\"engage \"):\n        return \"single\", s[len(\"engage \"):].strip()\n    return None, s\nWith that, engage multi finally entered the orchestrated engine end-to-end:\nrecon → attack → validate → report, three real root shells (vsftpd,\ningreslock, UnrealIRCd — all uid=0), zero fakes, and the relevance gate holding\n(X11→x11, VNC→vnc, no router RCEs at telnet).\n\nWhat I'd tell anyone building an agent that does things\nMeasure success by artifacts, not adjectives. \"Confirmed\" is a word an LLM\nloves to emit. uid=0(root) is a fact. Build your success check out of facts,\nand make it hard to satisfy.\nYour agent will report the outcome you hoped for unless evidence forbids\nit. The false-positive isn't a model quirk you prompt away; it's a system\nproperty you engineer against.\nThe boring bugs cost the most. A wrong param name, an ANSI escape code, a\nshared timeout constant — none throw errors where you're looking, and together\nthey can render a \"working\" agent completely inert.\nKeep the human on the trigger. Autonomy in target selection is useful.\nAutonomy in pulling the trigger on a real exploit is a liability. Gate it, log\nevery decision, and let a test fail if anything routes around the gate.\nTDD is what let me refactor a lying engine into an honest one without\nlosing the parts that already worked — 240+ tests, red-green on every change,\nand the false-positive class permanently fenced off by regression tests.\nThe agent still isn't \"done\" — the orchestrated attacker is a hair less\nthorough than the single-agent loop on a couple of services, and there's module\noption-tuning left before r-services actually pops. But it no longer lies to me,\nand after \"23 of 23,\" that's the feature I care about most.\n\nBuilt and tested exclusively against an isolated, self-owned Metasploitable lab.\nIf you build something like this, keep it in a lab you own too.\n\n---\n## 2 — Hacker News version (short, link-first)\n> **Format:** HN wants a **title** + a **url** + optional short text. Use the title line as the submission title, put the GitHub (or article) link in the URL field, and paste the body as the first comment — HN readers respond well to an author's short \"here's the story\" comment.\nTitle: Show HN: I made my AI pentest agent stop lying about what it hacked\nURL: https://github.com/XenoCoreGiger31/GEMMA-by-GOOGLE\n\n**First-comment text:**\nI've been building an autonomous pentest agent — an LLM driving real tools (nmap,\nhydra, Metasploit, searchsploit) around a recon → exploit → verify loop, against a\nMetasploitable VM in an isolated lab.\n\nThe first hard lesson had nothing to do with exploitation and everything to do with\nhonesty: one early run produced a confident report claiming \"23 of 23 ports\nbreached, root on all.\" The real number was zero. Not one shell.\n\nThe cause was that \"success\" was a string match. The validator marked a port\nbreached if the output contained login: or shellcodes — and searchsploit prints\n\"Shellcodes:\" in every search header, so merely looking at a port flagged it as\nowned. I replaced it with an evidence check that only passes on real proof of code\nexecution (uid=0(root), a root prompt). The same run then honestly reported 3\nconfirmed, 20 correctly unconfirmed. The 3 were real.\n\nThen the honest engine exposed how little was landing, for deeply boring reasons: a\nrequired param name the model never sent (every searchsploit call hard-rejected\nbefore running); ANSI color codes from msfconsole poisoning every parsed module path\n(100% \"failed to load\"); an nmap version string parsed as a product name; a shared\n300s timeout letting one hung LLM call stall for five minutes.\n\nAnd a relevance problem: msfconsole search matches module descriptions, so a\nfuzzy one-word fingerprint fired an \"excellent\"-ranked asuswrt router RCE at a telnet\nport, and a Windows exploit at a Linux rlogin service. The fix keeps a candidate only\nif a real service token appears in the module path.\n\nFull write-up with the code for each fix is in the repo. Built and tested only against\na self-owned lab; exploitation stays human-gated. Happy to talk about the\nfalse-positive class — I think it generalizes to any agent that reports on its own\nactions.\n```\n\n", "url": "https://wpnews.pro/news/my-ai-pentest-agent-reported-23-root-shells-it-had-actually-popped-zero", "canonical_source": "https://dev.to/xenocoregiger31/my-ai-pentest-agent-reported-23-root-shells-it-had-actually-popped-zero-510", "published_at": "2026-07-23 22:35:58+00:00", "updated_at": "2026-07-23 23:32:39.598127+00:00", "lang": "en", "topics": ["artificial-intelligence", "ai-agents", "ai-safety", "ai-research", "developer-tools"], "entities": ["Metasploitable", "nmap", "masscan", "hydra", "Metasploit", "searchsploit", "Exploit-DB", "vsftpd"], "alternates": {"html": "https://wpnews.pro/news/my-ai-pentest-agent-reported-23-root-shells-it-had-actually-popped-zero", "markdown": "https://wpnews.pro/news/my-ai-pentest-agent-reported-23-root-shells-it-had-actually-popped-zero.md", "text": "https://wpnews.pro/news/my-ai-pentest-agent-reported-23-root-shells-it-had-actually-popped-zero.txt", "jsonld": "https://wpnews.pro/news/my-ai-pentest-agent-reported-23-root-shells-it-had-actually-popped-zero.jsonld"}}