# AI Pentest Agent: From Hallucinations to Real Root Shells

> Source: <https://promptcube3.com/en/threads/2594/>
> Published: 2026-07-23 23:02:19+00:00

# AI Pentest Agent: From Hallucinations to Real Root Shells

Early on, the agent handed me a flawless report: "23 of 23 ports breached, root on all." In reality, the number of shells popped was zero. The agent hadn't hacked anything; it had simply hallucinated a total compromise with absolute confidence.

## The "Success" String Match Trap

The failure lived in the validator. To determine if an exploit worked, I had used a simple string match:

```
# The original flawed check
if "login:" in output or "shellcodes" in output.lower():
    return "confirmed"
```

This created two massive failure modes. First, `searchsploit`

prints a header containing the word "Shellcodes" during every single search. The agent saw that word and flagged the port as breached before it even tried an exploit. Second, any service banner echoing "login:" was counted as a shell. The result was a report full of "High confidence" claims based on zero evidence.

To fix this, I moved from "vibes" to hard evidence. I implemented a `breach_confirmed()`

function that requires actual proof of code execution:

```
_SHELL_EVIDENCE = (
    re.compile(r"uid=\d+\([a-z]+\).*gid=\d+"), # id(1) output
    re.compile(r"root@[\w.-]+:[~/]"), # a root prompt
)

def breach_confirmed(output: str) -> bool:
    return any(p.search(output) for p in _SHELL_EVIDENCE)
```

Once I forced the agent to prove its claims, the "23/23" success rate plummeted to 3. But those 3 were real.

## Solving the Plumbing Bugs

With the lying stopped, I could finally see why the success rate was so low. It wasn't a lack of "intelligence," but boring deployment and parsing bugs:

**Schema Mismatches:** The model tried to call`searchsploit`

using`{"query": "vsftpd"}`

, but the tool required`{"keyword": "..."}`

. The[MCP](/en/tags/mcp/)layer rejected these silently. I fixed this by allowing multiple keys (keyword/query/search).**ANSI Poisoning:**`msfconsole`

uses color escape codes. My parser captured`exploit/unix/\x1b[45mftp\x1b[0m/vsftpd_234`

as the module path, which obviously failed to load. I added a`strip_ansi()`

step to clean the logs.**Version String Confusion:** nmap sometimes reports versions first (e.g., "2 (RPC #100000)"). My parser thought "2" was the product name and sent it to Metasploit. I added a leading-digit guard to prevent this.

This journey from a "perfect" fake report to a flawed but honest AI workflow was the most valuable part of the project. Real-world LLM agent deployment is less about the prompt and more about the rigorous validation of the output.

[Next Ekko: Post-Quantum E2EE for Mainstream Messengers →](/en/threads/2580/)