Over the last month, the number of Large Language Model (LLM) penetration testing and orchestration frameworks has exploded. Just a few weeks ago, ARTEX — an open source agentic penetration testing system that automates offensive security workflows — was the only framework available. In just the last three days, we have seen over 50 framework families attempt to scan Cloudflare customers globally. The number of Internet Protocol (IP) addresses seen running these frameworks has grown into the thousands and continues to increase daily. The proliferation of these frameworks has significantly changed the status quo in cyber defense: defenders are struggling to keep up with the volumetric nature of these autonomously guided attacks.
ARTEX was taken down by its creator Autumn-27 on October 8. (“The malicious exploitation of this tool runs entirely counter to its original intent,” they wrote in Chinese.) But in the month since, the ecosystem has exploded with new fragmented versions of it. There are now over 12,000 repositories, with several hundred unique forks containing lineage from the original repository. Additionally, hundreds of other similar agentic orchestration frameworks have been created since October 8. Daily requests of AI penetration testing (pentesting) and exploitation tools has grown by 30x since October 1. These tools have also begun chaining stolen customer credentials in some of the more complex attacks, such as lateral movement chaining throughout an environment.
We expect these frameworks to continue to proliferate, and every organization should be prepared to address this threat. To counter LLM-based pentest frameworks, we recommend that all organizations adopt what we call the Modern Security Model, which has six key pillars:
- Fortify the front gate : Implement a Web Application Firewall (WAF) with advanced rules such as rate limits.
- Security station in the foyer : Anomaly detection on all web applications your customers access.
- Identify phony visitors : Implement Bot Defense.
- Remove the back door : Implement Identity-Gated Access for all high-risk external and internal web applications.
- Inspect deliveries: Enforce validated schemas and explicit security controls.
- Neighborhood watch : Implement Threat Intelligence to gain visibility into attacks.
What is an LLM penetration and exploitation orchestrator? #
Open-source pentesting frameworks are able to orchestrate AI agents to carry out stages of the traditional Cyber Kill Chain at scale. Tools like ARTEX have the ability to quickly plug in existing Model Context Protocol (MCP) and connect traditional standalone pentesting tools in novel ways. Some even allow operator direction on what type of task to accomplish. These tools can map and enumerate environments, both from the inside and outside. ARTEX and its forks map assets and environments onto a graph, allowing multiple independent parallel agents to enumerate, verify, prototype, and attack at the speed of inference. These tools turn compromises into a hill-climbable problem, which agents are specifically good at.
The AI LLM pentesting tools that currently exist
We are currently tracking 50+ AI LLM agents. The table below shows the top five LLM penetration and exploitation orchestrators. Each is extremely popular (1k+ GitHub stars) and fully autonomous. The fully autonomous nature of these agents represents a doctrinal shift in offensive cyber capabilities:
- Human skills are no longer the limit: A harness performs every step of the OODA loop by observing the environment, orienting itself to prioritize valuable and/or vulnerable targets, decide on an action, and act autonomously.
- Speed increase: Having multiple parallel agents working towards a goal is a large shift compared to single operator or multiple operator augmented by AI.
- Scale: A single operator can target multiple organizations at the same time, greatly increasing their blast radius.
- Small mistakes are easily discovered: Autonomous frameworks relentlessly and systematically test exposed services, including their configurations.
| | Architecture | Planning model | Built-in capabilities/execution environment | Post-exploitation capabilities | | ARTEX (and forks) | Planner and parallel agents | Graph of goals, leads, and findings is linked to separate asset graph of target | Directly on host or container with network/root access, can install/develop tools as needed | Nothing built-in, requires user instruction | | LuaN1aoAgent | Supervisor agent, planner agent, and generic executor agents | Three graphs, with goals/tasks/blockers, evidence/hypothesises/exploits, and target/local | Container with network/user access, can develop tools as needed and install user packages | Built-in SSH and Chisel for tunneling and attempted reconnection after disruption | | Decepticon | Orchestrator agent dispatching specialist sub-agents | Graph of objectives with prerequisites | Kali container running as root, can develop tools as needed | Dedicated post-access, Active Directory and cloud pivoting skills | | Cairn | Dispatcher orchestrating work to existing AI harnesses (Codex, Claude Code, Pi) | Graph of facts and leads; new nodes trigger dispatcher | Kali container with root access or directly on host, can develop tools as needed | Prompt guidance and Kali tools | | PentestGPT-Agent | Supervisor agent and one generic executor agent | Task list with dependencies, goals, and evidence | Run on host, can develop tools as needed | Nothing built-in, requires user instruction |
Observations from the world's largest network #
As of November 12, ARTEX and its forks make up over 60% of currently known AI penetration testing orchestrators. The charts below help us understand how adoption of these tools has changed over the past six weeks. Since the removal of the original ARTEX repository on October 8, traffic has grown more than tenfold, and has increased more than 30x since October 1.
As demonstrated above, the takedown of ARTEX on October 8 (and widespread reporting of it) likely caused a hydra effect, where the removal of the original repository led to a massive increase in the number of forks and modifications which can be seen in the steep acceleration of unique instances around October 8.
Identity-chained LLM exploitation
Over the last eight weeks, our Cloudforce One team has been called into multiple incident response investigations involving ARTEX and ARTEX-derived LLM exploitation frameworks. We’ve seen firsthand how exposed credentials are being chained with LLM-based exploitation frameworks to gain the upper hand over defenders.
We observed multiple incidents in which a threat actor gained initial access to a hyperscaler environment and pivoted directly into the customer's Cloudflare environment. Root-cause analysis revealed a common credential hygiene failure. Attackers repeatedly compromised static, non-expiring tokens running inside CI/CD pipelines, which granted broad, cross-environment access across the organization's hyperscaler infrastructure. LLM frameworks greatly expand the likelihood of a client’s API key being laterally leveraged to take over other tenant environments.
ARTEX in action
As part of one incident response investigation, our team observed ARTEX gaining access to a compromised account-level API token previously used for CI/CD with no IP restrictions or rate limiting. Available evidence indicated that the compromise most likely originated through a third-party hyperscaler.
We observed that a large number of new Worker scripts had been created with distinct names such as artex-, artex-pgrecon-, artex-pgw-, and artex-do-. The naming of each Worker script correlated to distinct behaviors and actions it was taking, and was often postfixed by an alphanumeric identifier. We observed activity in excess of 69 requests per minute across multiple unique parts of their Cloudflare infrastructure and 400 database dump requests in an hour. We also observed attempted persistence mechanisms and attempts to validate session tokens for other third-party applications. While we believe that much of this activity was lightly guided by the human operator, this is just one example of the speed and breadth that ARTEX and similar frameworks can achieve post-compromise.
Our recommendation: the Modern Security Model #
Large Language Models increase the ability of a threat actor to rapidly iterate through the Cyber Kill Chain against a target. They do this by giving the actor the advantage of speed and validation. Each step of the Cyber Kill Chain can be validated and fed into the next stage of the attack. This level of orchestration is unparalleled and effectively vaporizes legacy cyber security models. A modern model must be used against these attacks.
To help explain this model further, let’s use the metaphor of protecting a house on an estate. Each section below outlines how to defend against the threat at each step.
Fortify the front gate: implement traffic control and a WAF with advanced rules
Before anyone even sets foot on your lawn, they have to reach the perimeter fence.
On an unprotected estate, an adversary can drive an endless fleet of decoy trucks straight into your driveway, completely blocking the entrance and smashing through the barriers (Layer 3/4 volumetric floods and direct IP probing).
With perimeter gate security in place, all incoming traffic is routed through massive, distributed outer gates miles away from the main house. Rogue delivery trucks are filtered out to the highway, rogue utility tunnels trying to wiretap your property are severed, and only legitimate foot traffic ever reaches your home address.
Network trafficcontrol serves as an important role in stopping volumetric attacks.
A security station in the foyer: anomaly detection
Once visitors arrive at the main entry, they meet a strict security station at the foyer.
Automated prowlers don't just kick the door; they disguise themselves. They attempt weird ways to get in the door — stuffing paper into the lock, trying variations like //front-door//back-room/ to trick the mechanism into swinging open.
The security guard at the door enforces clear rules:
- You cannot knock 500 times in five seconds.
- You cannot disguise your path with trickery.
- You cannot attempt to roll a massive flatbed cart through the front hall to haul away the house's internal blueprints (diagnostic heap dumps). Suspicious behavior is turned away immediately.
- Implement security keys and hardware based FIDO keys.
Web Application Firewall (WAF) serves as a key defense mechanism in stopping application exploitation through rate limits and anomaly detection.
Identify phony visitors: implement Bot Defense
A sophisticated intruder rarely looks like a masked burglar; they disguise themselves as ordinary visitors. But automated agents, headless browser scrapers, and AI reconnaissance tools don't act like real people.
They don't blink, their posture is unnervingly rigid, their cadence is mechanical, and their "identification" is often a sloppy imitation of a real visitor badge.
Bot Defense and challenges act as behavioral specialists watching the crowd through one-way glass. It flags the unnatural cadence, catches the synthetic micro-behaviors, and quietly escorts the machines off the property before they can touch a single doorknob.
Remove the back door: implement Identity-Gated Access and micro-segmentation
Many homes historically were built with side doors or a cellar hatch (internal admin panels, /actuator diagnostic endpoints, and remote gateways). Even if locked, a prowler could stand on the sidewalk, shine a flashlight on the cellar door, and start picking the lock.
Zero Trustapplication access does something radical: it bricks up the doorway from the outside and restricts which rooms can be entered.
To anyone wandering the perimeter, the cellar door does not exist. There is no lock to pick, no handle to rattle, and no light on inside. The only way in is with an escort who checks your biometric badge, confirms your device has a clean bill of health, and steps you directly into the specific room you need — leaving the rest of the house out of view.
For those who do enter, you have to ensure they cannot breach rooms next to the room they enter. Micro-segmentation is a highly effective strategy at stopping lateral movement. It limits compromised credentials from breaching production, test, and development systems.
Inspect deliveries: enforce validated schemas and explicit security controls
You receive daily supply deliveries to your estate — groceries, mail, maintenance parts. In digital terms, this is your incoming API traffic, which must be inspected and verified.
Intruders often attempt to bypass controls by sending unvalidated payloads or forged delivery manifests meant for restricted internal systems.
The porch protocol mandates thorough inspection and schema validation for all traffic:
- Every incoming request must undergo OpenAPI schema validation to ensure it strictly conforms to expected definitions and structures.
- Drivers and couriers must present validated mutual TLS (mTLS) certificates to establish trusted, authenticated connections.
- Any request failing inspection or missing valid credentials is rejected immediately at the boundary.
- Default-Deny Posture: Every positive security implementation ends with an explicit deny-all rule at the bottom of the ruleset.
Neighborhood watch: implement threat intelligence to gain visibility into attacks
Even with ironclad walls, an isolated property can be vulnerable if a syndicate invents a brand-new infiltration technique across town.
Threat intelligence acts as your warning network with neighboring estates. When a rogue actor sets up a new command-and-control safehouse down the road or begins deploying dynamic tunneling vans in your district, you receive an early dossier. Your estate updates its watchlists, recalibrates its sensors, and slams the gate shut before the actor ever turns onto your street.
How to get real-time help #
ARTEX and similar LLM-based frameworks pose a significant challenge to legacy cyber defense models. Even the historical defense-in-depth model — which emphasizes a classic network DMZ — falls short in today's complex architectures. Tomorrow's attacks will only advance as orchestration tools become the new normal.
Securing legacy equipment and on-premises devices requires an identity-gated approach that logically makes these devices impossible to reach on the Internet. Beyond identity, slowing or making lateral movement more noisy is the goal of micro-segmentation. This architecture emphasizes a containment-first approach to security, which is essential when users are consistently losing their keys. When combined, the elements of the Modern Security Model are the answer to this vexing problem.
If you are facing an active security incident and need the Cloudforce One team on the ground to help respond to the incident or require threat intelligence support, please contact us immediately. If you need help implementing the Modern Security Model outlined here, please contact our Solutions Engineering team.