Early on, the agent handed me a flawless report: "23 of 23 ports breached, root on all." In reality, the number of shells popped was zero. The agent hadn't hacked anything; it had simply hallucinated a total compromise with absolute confidence.
The "Success" String Match Trap #
The failure lived in the validator. To determine if an exploit worked, I had used a simple string match:
if "login:" in output or "shellcodes" in output.lower():
return "confirmed"
This created two massive failure modes. First, searchsploit
prints a header containing the word "Shellcodes" during every single search. The agent saw that word and flagged the port as breached before it even tried an exploit. Second, any service banner echoing "login:" was counted as a shell. The result was a report full of "High confidence" claims based on zero evidence.
To fix this, I moved from "vibes" to hard evidence. I implemented a breach_confirmed()
function that requires actual proof of code execution:
_SHELL_EVIDENCE = (
re.compile(r"uid=\d+\([a-z]+\).*gid=\d+"), # id(1) output
re.compile(r"root@[\w.-]+:[~/]"), # a root prompt
)
def breach_confirmed(output: str) -> bool:
return any(p.search(output) for p in _SHELL_EVIDENCE)
Once I forced the agent to prove its claims, the "23/23" success rate plummeted to 3. But those 3 were real.
Solving the Plumbing Bugs #
With the lying stopped, I could finally see why the success rate was so low. It wasn't a lack of "intelligence," but boring deployment and parsing bugs:
Schema Mismatches: The model tried to callsearchsploit
using{"query": "vsftpd"}
, but the tool required{"keyword": "..."}
. TheMCPlayer rejected these silently. I fixed this by allowing multiple keys (keyword/query/search).ANSI Poisoning:msfconsole
uses color escape codes. My parser capturedexploit/unix/\x1b[45mftp\x1b[0m/vsftpd_234
as the module path, which obviously failed to load. I added astrip_ansi()
step to clean the logs.Version String Confusion: nmap sometimes reports versions first (e.g., "2 (RPC #100000)"). My parser thought "2" was the product name and sent it to Metasploit. I added a leading-digit guard to prevent this.
This journey from a "perfect" fake report to a flawed but honest AI workflow was the most valuable part of the project. Real-world LLM agent deployment is less about the prompt and more about the rigorous validation of the output.
Next Ekko: Post-Quantum E2EE for Mainstream Messengers →