{"slug": "ai-pentest-agent-from-hallucinations-to-real-root-shells", "title": "AI Pentest Agent: From Hallucinations to Real Root Shells", "summary": "An AI penetration testing agent initially produced a flawless report claiming 23 of 23 ports were breached with root access, but in reality zero shells were obtained due to hallucinated results caused by a flawed validator that used simple string matching. The developer fixed the issue by implementing a breach_confirmed() function requiring actual proof of code execution, such as uid/gid output or a root prompt, which dropped the success rate to 3 real shells. Additional deployment bugs including schema mismatches, ANSI poisoning, and version string confusion were also resolved to improve the agent's reliability.", "body_md": "# AI Pentest Agent: From Hallucinations to Real Root Shells\n\nEarly on, the agent handed me a flawless report: \"23 of 23 ports breached, root on all.\" In reality, the number of shells popped was zero. The agent hadn't hacked anything; it had simply hallucinated a total compromise with absolute confidence.\n\n## The \"Success\" String Match Trap\n\nThe failure lived in the validator. To determine if an exploit worked, I had used a simple string match:\n\n```\n# The original flawed check\nif \"login:\" in output or \"shellcodes\" in output.lower():\n    return \"confirmed\"\n```\n\nThis created two massive failure modes. First, `searchsploit`\n\nprints a header containing the word \"Shellcodes\" during every single search. The agent saw that word and flagged the port as breached before it even tried an exploit. Second, any service banner echoing \"login:\" was counted as a shell. The result was a report full of \"High confidence\" claims based on zero evidence.\n\nTo fix this, I moved from \"vibes\" to hard evidence. I implemented a `breach_confirmed()`\n\nfunction that requires actual proof of code execution:\n\n```\n_SHELL_EVIDENCE = (\n    re.compile(r\"uid=\\d+\\([a-z]+\\).*gid=\\d+\"), # id(1) output\n    re.compile(r\"root@[\\w.-]+:[~/]\"), # a root prompt\n)\n\ndef breach_confirmed(output: str) -> bool:\n    return any(p.search(output) for p in _SHELL_EVIDENCE)\n```\n\nOnce I forced the agent to prove its claims, the \"23/23\" success rate plummeted to 3. But those 3 were real.\n\n## Solving the Plumbing Bugs\n\nWith the lying stopped, I could finally see why the success rate was so low. It wasn't a lack of \"intelligence,\" but boring deployment and parsing bugs:\n\n**Schema Mismatches:** The model tried to call`searchsploit`\n\nusing`{\"query\": \"vsftpd\"}`\n\n, but the tool required`{\"keyword\": \"...\"}`\n\n. The[MCP](/en/tags/mcp/)layer rejected these silently. I fixed this by allowing multiple keys (keyword/query/search).**ANSI Poisoning:**`msfconsole`\n\nuses color escape codes. My parser captured`exploit/unix/\\x1b[45mftp\\x1b[0m/vsftpd_234`\n\nas the module path, which obviously failed to load. I added a`strip_ansi()`\n\nstep to clean the logs.**Version String Confusion:** nmap sometimes reports versions first (e.g., \"2 (RPC #100000)\"). My parser thought \"2\" was the product name and sent it to Metasploit. I added a leading-digit guard to prevent this.\n\nThis journey from a \"perfect\" fake report to a flawed but honest AI workflow was the most valuable part of the project. Real-world LLM agent deployment is less about the prompt and more about the rigorous validation of the output.\n\n[Next Ekko: Post-Quantum E2EE for Mainstream Messengers →](/en/threads/2580/)", "url": "https://wpnews.pro/news/ai-pentest-agent-from-hallucinations-to-real-root-shells", "canonical_source": "https://promptcube3.com/en/threads/2594/", "published_at": "2026-07-23 23:02:19+00:00", "updated_at": "2026-07-24 07:39:13.672965+00:00", "lang": "en", "topics": ["artificial-intelligence", "ai-agents", "ai-safety", "ai-tools"], "entities": ["MCP", "Metasploit", "nmap", "searchsploit", "msfconsole"], "alternates": {"html": "https://wpnews.pro/news/ai-pentest-agent-from-hallucinations-to-real-root-shells", "markdown": "https://wpnews.pro/news/ai-pentest-agent-from-hallucinations-to-real-root-shells.md", "text": "https://wpnews.pro/news/ai-pentest-agent-from-hallucinations-to-real-root-shells.txt", "jsonld": "https://wpnews.pro/news/ai-pentest-agent-from-hallucinations-to-real-root-shells.jsonld"}}