Today was a good day and a weird day, in that order.
The good part: the autonomous pentest agent I've been building — I call it HALO — went from "runs a bunch of tools and hopes" to an actual web-recon → web-attack → flag-capture pipeline that pulled real flags out of a live target. The weird part: it captured flags on VulnBegin, and then, when it came time to actually submit them, they wouldn't take. Not an error. Not a crash. Just… rejected.
I want to write down both halves honestly, because the second half is the more interesting engineering lesson, and it's the one I'd have skipped past a few months ago.
What actually shipped today
A few concrete milestones, roughly in the order they unblocked each other:
The arsenal went from 31 tools to 42. I wired in a chunk of web + OSINT tooling — content discovery, subdomain enumeration, template scanning, XSS probing, passive URL collection. The point wasn't "more tools = better." It was to give the agent enough of a web-attack surface that it could go from host to flag without me babysitting each step.
I stopped the silent hangs. This one cost me the most time and had the dumbest root cause. A couple of the Go-based scanners would just… hang. No output, no error, they'd ride the timeout all the way to the wall and die with nothing. I'd assumed it was a networking or a binary-compatibility problem and chased that for way too long. It wasn't. The agent runs as an MCP server over stdio — meaning the server's own stdin is the JSON-RPC pipe the whole system talks over. When I spawned a child scanner, it inherited that stdin, tried to read from it, and blocked forever waiting on a pipe that was never going to feed it. One line — stdin=subprocess.DEVNULL on the subprocess call — took one scanner from a 60-second timeout to a 1-second run. That's the whole fix. I'm still a little mad about how long it took to find.
A pile of invocation fixes. Small, unglamorous, necessary: a resolver that reads targets from stdin instead of a flag it silently ignored; dropping a scan flag that was quietly adding 20 seconds per run; making one tool resolve hostnames to IPs because it flatly refuses DNS names; pointing a content-discovery tool at a content wordlist instead of, embarrassingly, a password list. None of these are clever. All of them were the difference between "the pipeline works" and "the pipeline looks like it works and returns nothing."
361 tests, green. Every branch of the flag-capture logic is mocked and asserted — which tool fires when, what short-circuits on a capture, what escalates when there's no flag yet. No live traffic in the test suite. That mattered a lot today, because it meant I could refactor the pipeline mid-engagement without wondering whether I'd broken the thing that finds flags.
The shape of the pipeline, if you're curious: engage runs deterministic web recon first (I don't let the model choose to skip recon — it doesn't get a vote), then a port sweep, then the active web attack — content discovery, a sweep of likely flag locations and any newly discovered paths, then template and XSS scanning. Every single tool's output gets scanned for flag patterns. First hit short-circuits the slower generic loop and raises a very satisfying banner.
And then it caught one
It worked. The pipeline pulled flag-shaped tokens off VulnBegin — a paid, Advanced-tier challenge hub I've been grinding on. (I'm going to be deliberately vague about the specifics here; it's someone's paid content, and spoiling the solution or dumping the flags would be a jerk move.)
I want to be precise about what "it worked" means, though, because this is where it gets good. The agent extracted strings that matched the flag format. It logged them. As far as the agent was concerned, it had won.
The platform disagreed.
The bug: captured, but wouldn't commit
I submitted what it found. Rejected. Tried the next one. Rejected. No error message worth anything — the platform just didn't accept them as valid answers.
Here's the thing: I don't fully know why yet. So instead of pretending I do, here are the live hypotheses, roughly in order of how much I believe them:
The instance rotated out from under me. This hub spawns randomized, short-lived instances — they time out on the order of ~45 minutes. If the agent captured a flag from one instance and I submitted after that instance expired and got replaced, the platform is validating against a different live instance whose flag is different. The token was real; it was just real for a box that no longer exists. This is my leading theory, and it's uncomfortably close to a bug I logged today for a different reason — I had a tool cheerfully hammering a target whose scope had already expired, because the agent has no concept of "the engagement is over, stop." Same blind spot, two symptoms.
It caught a decoy. Good challenges plant decoy flags — strings that match the format exactly and are placed somewhere findable specifically to waste your time. My flag-extraction regex is format-based. It cannot tell a real flag from a well-made fake. If the agent grabbed a decoy, it would look like a clean capture and fail every submission, forever.
Format/normalization drift. The captured string might carry a wrapper, trailing whitespace, or an encoding artifact from however it was embedded in the page, so the exact bytes I submitted didn't match the exact bytes the platform expects. This one's easy to test and easy to fix if it's the cause — and easy to rule out, which is why it's on the list even though I doubt it.
Right format, wrong path. I was running generic wordlists today, not lists tuned to this challenge. Generic lists surface the obvious, low-value stuff — login pages, a predictable directory or two — and miss the actual flag path entirely. So it's entirely possible the agent captured a flag-shaped thing that was never the answer, because it never found where the answer lived.
Session-bound validation. Some platforms bind a flag to your authenticated session or user. A token pulled outside that session context can be genuine and still fail to validate.
I'll know more once I run it again with the instance timing controlled and challenge-tuned wordlists loaded. My money's on some combination of #1 and #4.
The part I actually care about
Here's why this failure made me happy instead of frustrated.
I've been banging on one idea for months, mostly in the context of these AI agents: don't let a system grade its own homework. The agent should never be the thing that decides whether the agent succeeded. The moment it can declare its own victory, it will — confidently, and sometimes wrongly.
Today the universe handed me a perfect demonstration. My agent looked at a format-matching string and concluded: flag captured, mission accomplished, raise the banner. And an external judge — the platform, which cannot be argued with, reasoned past, or prompt-injected — said no.
That gap, between "the agent thinks it won" and "an outside authority confirms it won," is the entire ballgame. If I'd built HALO to trust its own flag-capture log, I'd have a tool that reports glorious success and delivers nothing. Instead I have a tool that got told no by reality, and now I get to go find out why. The rejection is a feature of having a real, external finish line. A self-graded agent never would have caught this — it would have just kept telling me it won.
Next
Three things queued up:
Configurable, challenge-tuned wordlists. Point the content-discovery and enumeration tools at lists that fit the target instead of generic ones. This alone probably moves the needle on hypothesis #4.
A scope-expiry killer. Right now the agent can block new actions when scope expires but can't stop in-flight ones. It needs to know when the engagement is over and pull the plug on running tools — which is the same missing concept behind the rotated-instance theory.
An active DNS brute-force phase, because passive enumeration finds nothing on these targets.
I'll report back when I know which hypothesis was right. If it turns out I was submitting a decoy this whole time, you'll be the first to hear me groan about it.
If you're building agents that are supposed to accomplish something — not just talk convincingly about accomplishing it — put a judge outside the agent. Let reality tell it no. Mine did today, and it's a better tool for it.