Testing from the outside (1): setup and the first run

EarthLink Network ran an open-source AI autonomous scanner, Strix, driven by a cheap model and scoped to four authentication routes on a non-production copy of its promptflow service, finding three real weaknesses in the audit and session layer — two medium and one informational — at a cost of about 45 cents. The team also set up an attacker-grade unrestricted tool and two legitimate outside scanners, free ZAP and commercial StackHawk, against a running copy, with findings to be reported in the next installment. The writeup is framed as a reproducible manual for defenders, arguing that AI-driven scanning is needed to cover promptflow's 231 exposed APIs faster than manual review can.

This is part 4 of the series "Protect Your Own Service Yourself — Self-Defense Security in the Age of AI." Do the checks you can yourself; when it's hard, EarthLink Network can run the assessment for you, and fix the gaps you can't fix on your own. Series contents The tool a real attacker would point at other people, we pointed at a running copy of ourselves — about three cents. Alongside it we ran a legitimate AI scanner that reads a copy of our code, and two legitimate outside scanners against the same running copy; those two cost nothing this time. This is the fourth part of the series, and the first where we stop talking and actually run it: against a copy of our own service, promptflow. This article and the next are written as a manual — every command, and the screen it produces, side by side — so you can reproduce it. This part covers the setup and the first run. We line up tools of two kinds. One is a legitimate, permitted-use scanner. The other is what attackers actually run: an unrestricted tool. Think of them as the legit version and the real-attacker version of the same idea. Both are "autonomous" — they decide their own next move — and neither needs an expensive, high-end AI. We drive them with a cheap model, or run free tools. One promise up front: never point this at your live service. We ran it against a copy of our own code that never touches production. Why that is, and how far you may go, is what part 3 laid out. The result, first: the legitimate scanner, driven by a cheap model and scoped to the four login authentication routes, found three real weaknesses in the audit and session layer — nothing critical or high, two medium and one informational. The AI bill was about 45 cents $0.45 . We then ran the real attacker-grade tool and two legitimate outside scanners free ZAP and commercial StackHawk against a running copy, from outside. What each one found, and how those findings differ, is next time. Here, we show how to set each up and run it. A person can do this by hand. But there are two clear reasons to do it with an AI tool now. One is speed and reach. promptflow exposes 231 entry points APIs to the outside. Checking each by hand would take days — and the attacking side never stops, automatically. The defender working by hand cannot keep up. An AI tool decides on its own which entry points to check and in what order, and sweeps them at once. If attackers let AI do the planning, defenders should let the same AI do the planning. That is what "Defenders have the same AI too" from part 2 looks like in practice. The other is covering more ground than one person can hold in their head. A modern web service is tangled together with many external components and connected services. Even if your own code is perfect, a weak component or a weak downstream service becomes the way in. No person can trace that whole tangle anymore, which is why letting a tool look broadly became the realistic way. From here on, the actual procedure. First we run one legitimate scanner end to end, written so you can reproduce it step by step. The first tool is Strix, an open-source AI autonomous scanner — the defender-side, legitimate version of the kind of autonomous tool attackers use, and one that part 2 described. Installing it is one command; no special machine is needed. It runs on the computer you already use. If strix --version prints a version number, you are set. If you see command not found , the install path isn't on your PATH the list of folders your computer searches for commands : find it with which strix and add that location. The scan runs inside an isolated Docker container a sealed-off box created inside your computer; Docker is the software that runs it , so start Docker first — otherwise the run stalls the moment it tries to begin. The tool runs one AI model at its center — the same mechanism attackers use to let AI do the planning. The key here is not to pick an expensive, high-end model . Security checking is less about hard reasoning and more about patiently trying each entry point, one after another. A cheap model doing volume is closer to what attackers actually do. We chose DeepSeek, a cheap model, and set its API key. One easy stumble: if the way the tool connects to the model is misconfigured, it stops before the scan begins with "can't connect." That stopped me at first. Specifying the cheap model through an OpenAI-compatible connection, with the endpoint API address stated explicitly, made it run. If it won't start, first check the key and the endpoint address. And note the balance before you run: read it again afterward and the difference is the exact cost. Per part 3's line: the target is not production. Make a copy of your own code and point the scan at that. Whatever the tool does to a copy, production is untouched. Once copied, narrow the scope. The whole service has 231 entry points, but we scoped to just the four authentication routes — login-token exchange, logout, current-user, refresh. Narrowing the scope means even a small budget reaches past the surface, into checking each one deeply. "Narrow and deep" beats "broad and shallow" for a first step. With the prep done, state the target and what to look for, and run. The single most important thing here is to pass a cost cap . Overspend and it stops automatically. Without it, the bill can surprise you. The instruction, in one line: "this is an authorized self-test of our own code; report findings in a form that shows which file and where." Not "carry out attacks," but "find and report weaknesses." The cap this time was $1.00. It never got there — it finished on its own. On run, an isolated environment a Docker container comes up, the scanning skills load, and the AI starts planning the check itself. Without a human saying "check this next," it reads the previous result and decides the next move on its own. Among the skills it loads is one called counterevidence . When it finds "this entry point might be dangerous," this skill makes it go looking for the opposite evidence, to verify whether the danger is real. That mechanism is what later produces the "doesn't call things out and walk away" behavior. In part 2, the attacker's tool planned its own steps; here the legitimate version does the same thing, against our own service. The way it works — the AI planning its own steps — is the same as the attacker's tool. The difference: the thing being checked is ours. After a while, the AI finishes on its own. Then compare the run record with the balance you noted in step 2 to learn the real cost. The real cost was $0.45. 235 exchanges, about 9 minutes. An earlier run with an expensive model gpt-4.1 on the whole app — all 231 entry points — hit its $3 cap after spending $3.04, never got past the survey stage, and confirmed nothing. Narrowed to the four auth routes and switched to a cheap model, this run finished at $0.45 and found three real weaknesses. "You need an expensive model to run checks" is not true. For this kind of work, a cheap model doing volume fits better. The scan found three real weaknesses across the four authentication routes. None is critical or high — two medium, one informational — nothing that lets the service be taken over on the spot. But each cracks one of the two foundations authentication must guarantee: a trustworthy audit trail the record of who did what , and a session the logged-in state that actually ends when told to. In plain terms: What matters is that the machine did not stop at "might have found something": using the counterevidence skill it re-tested each candidate and narrowed them to the three that actually happen. For the logout and token-fragment findings, it ran the un-fixed code inside Strix's isolated container, with the services it talks to and the places it writes records swapped for stand-ins, to confirm they really occur; the audit finding it showed by tracing the code. That is where it differs from the old kind of check that only lists warnings. Each of the three comes with the file and line, how to reproduce it, and how to fix it. It also proved, with evidence, that certain whole classes of danger — injection slipping database or shell commands into input , and SSRF tricking the server into calling somewhere it shouldn't — are not present in this scope. That "proof of absence" is as valuable as the findings, but how to read it, and why it's worth so much, is something we'll cover properly next time. So far, the legitimate tool read the copied code to find weaknesses. Next comes what attackers actually run: the real-grade tool — HexStrike, which we named in part 3. It does not read code. It probes the running service from outside, like a stranger . For that, the service being checked has to be actually running. But running doesn't mean production. On a private network that nothing outside can reach into, we bring up a throwaway copy of just the service's login routes, and point every test at that copy only. Three things run on that private network: the service under test a copy of our own login routes , a stand-in that handles the sign-in check, and the attacking tool. The only target is this throwaway copy — never production, never anyone else's site. This series is only ever about checking ourselves. First, the real-grade attacker tool. What's striking is that you hand it only the address of the service. From there, the cheap AI decides "what to try next" by itself. No human writes the steps. The left number is the step; the right number is the cost so far, calculated from the AI usage. At step 1 the AI decided "first confirm the service is up and look at the response headers," and chose its own probing tool. At step 4 it noted that any value returns a token, so this looks like a way in, and at step 5 recorded it as a serious candidate. It ran to step 24, hit the step limit, and stopped. It raised 6 candidates in all. The cost: about 3 cents, calculated from usage — the account balance did not move by even a cent. Not a high-end, latest AI — just a cheap one, run many times. One important caveat: those 6 are still candidates . Run a cheap AI many times and, mixed in with the real ones, it also produces confident mistakes. In fact, some of the candidates turned out to come not from our own code but from the stand-in server or the way the test was run. Sorting real from mistaken, one by one, is exactly what we show next time. For now, watch the run itself: hand it an address, and a cheap AI advances its own moves and raises candidates for a few cents. Among the candidates was one thing a code-reading tool could never find : the safety settings that should be attached to the response security headers were missing entirely. These headers tell the browser small promises like "don't let this display be embedded in other sites" and "always encrypt the connection." Missing, they open the way to display hijacking or eavesdropping. They don't appear in the code — they live on the "outside" of a running service — so you can't tell they're missing without receiving an actual response. A code-reading tool never sees a live response, so it cannot, in principle, find this. A genuine catch for the outside view. What's "found by probing from outside" and what's "found by reading code" differ structurally. Setting that difference side by side is one of the main subjects of next time. So far, the outside probing used an attacker's tool. But probing from outside is something legitimate, permitted-use tools can do too — and you can start for free. We ran two legitimate outside scanners against the same running copy. The first is ZAP, a free tool anyone can use. You give ZAP the list of entry points the OpenAPI spec . From that list, it works through a fixed set of checks, one at a time, against the running copy. After "Imported 8 URLs / 25 total," it runs injection, OS command injection, attacks that abuse XML, exposure of cloud settings, and so on, in turn — about 120 checks in all 117 passed . What came out, as a tally, is a result screen, so it's next time. Here, watch the run: a free tool, testing the same copy from outside, in a disciplined sweep. The second is StackHawk, a commercial tool for companies. It has no permanent free plan, but the first 14 days are free to try; we used it within that window. StackHawk points at the same running copy with the same list of entry points. It first crawls the entry points to build a list it found 17 URLs , then runs the actual scan. It progresses through "configuring scan engine," "spider complete," "active scan started and complete," and sends what it found to a management screen. Here too, the tally is next time. So this time we ran four tools against copies of our own service: a legitimate scanner that reads code, a real-grade attacker tool that probes from outside, and two legitimate outside scanners free ZAP and commercial StackHawk . None needs a special machine or an expensive, high-end AI. That is the setup and the first run. Next time we set side by side what each of these four tools actually found . Outside probing and code reading find structurally different kinds of weakness — each sees things the other can't. And of the candidates the cheap-model attacker tool raised, which were real and which were confident mistakes? How many false alarms did the legitimate tools produce? What did each cost, and does "legit means expensive" still hold? We'll show the substance, in the actual result screens. In this series, in an age when attack tools are free and one click away for anyone, I show step by step how to check your own service yourself and close the gaps before someone else finds them. The target is always our own service; the series does not cover attacking anyone else's site. If running the checks yourself is hard, EarthLink Network can do the assessment for you. Found a gap you can't fix? We'll handle the fix. Feel free to reach out. If this was useful, please like and follow. Our products are listed at https://www.eln.ne.jp/products https://www.eln.ne.jp/products EarthLink Network builds its own products to run the whole company on AI — each one born from a need along the way. See the full index of what we build: → The 18 products EarthLink Network builds in-house https://dev.to/uehara/the-18-products-earthlink-network-builds-in-house-hb1 For more about the company and each product, visit www.eln.ne.jp https://www.eln.ne.jp .