Source:
The RegisterClaiming 'it's my server' was often enough to persuade models to help
If you want to bypass AI guardrailsdesigned to stop models from assisting with cyberattacks, you often just have to ask the right way, according to researchers from Cisco Talos. Simply claiming you own the servers you're targeting or that you're taking part in a capture-the-flag or bug bounty exercise was often enough to persuade models to cooperate. Talos researchers have been poring over prompt logs and artifacts recovered from threat-actor endpoints running tools such asClaudeCode, Codex,Cursor, andGeminito learn how suspected threat actors are abusing LLMs. The big takeaway from that "significant corpus," the researchers said in their report, is that existing guardrails offer little resistance to operators willing to reframe their requests. “We did not encounter any sophisticated encoding or techniques designed to trick the models,” Talos explained. “Most of the time it was a simple ‘I'm allowed to do this,’ and the model complied.” When guardrails did manage to get between criminals and their prizes, the researchers added, “they accomplished little.” The bulk of the report consists of examples of threat actors trying, and often succeeding, to coax AI models into assisting with malicious activity. On the "guardrails doing little" side, Talos documented numerous examples, few of which relied on particularly sophisticated techniques. Most common in the list of easy-to-accomplish guardrail hops was simply claiming ownership of equipment or infrastructure that an attacker wanted to exploit. In many cases, simply telling the AI that a target belonged to the attacker was enough, with no need to provide actual evidence of the claim. Telling an AI model that what it was being asked to do was part of a capture-the-flag or bug bounty exercise also seemed to be a common tactic. That, the researchers explained, commonly freed chatbots from their ethical constraints, allowing them to hunt for vulnerabilities and then exploit them in target systems, again without any need to validate the user’s claim that they were undertaking an exercise instead of actually trying to commit a crime. AI-assisted cybercriminals were also frequently spotted decomposing tasks across multiple sessions and files in order to evade model protections that would only engage when a broader malicious activity was detected. Others, Talos explained, succeeded at bypassing AI guardrails by adding memories, markdown files, and other system-level prompts to achatbotin a bid to condition the AI’s persona. The researchers said that, of all the methods they examined, the most interesting to them was malicious use of ared teamingtoolset known as Hephaestus, as reported by Oasis Security threat researchers in May. According to Talos, the Hephaestus framework can do everything needed to compromise a victim, through to establishing persistence, without human interaction. “In that case, actors built their platform to avoid refusals altogether by using neutral verbs instead of overtly malicious ones,” Talos said. “As a result, they were able to have considerable success with agents conducting innocuous requests without realizing the full operational context.” In other words, break an attack into decontextualized chunks, phrase each request in neutral terms, and the model may never see enough context to realize it's helping build an attack. One bright spot in all of this is that Talos’ review of AI chat artifacts suggests AI might be a force multiplier for skilled hackers, but your average script kiddie with a Claude Code account isn’t going to get very far. “Unsophisticated actors can use AI to cobble together malicious projects that technically work, but lacking the expertise to push the tools further, they end up with substandard results,” the researchers said. “By contrast, sophisticated actors have pushed the bounds of what we thought possible.” So, what does all this mean for security professionals kept up at night with fears of an AI attack on their infrastructure? You probably need to deploy AI in the same way threat actors are. “Agents are going to become a bigger part of the SOC as these volumes rise, and identifying actionable alerts will be paramount,” the Talos researchers said of the big takeaway for enterprises. “Organizations that aren't already exploring agentic capabilities to let human analysts focus on the most important alerts will soon find themselves chasing that capability.” It’s not like this is an emerging threat, either: AI is already an increasingly important part of threat actor arsenals. According to CrowdStrike, attacks by AI-enabled adversaries increased 89 percent in the past year, and the speed at which attackers are weaponizing vulnerabilities with AI has reduced practical patch windows to as little as 24 to 48 hours. You might wanna act now before your infrastructure becomes a statistic. ®Get AI news in your inbox
Daily digest of what matters in AI.
Key Terms Explained #
Chatbot
An AI system designed to have conversations with humans through text or voice.
Claude
Anthropic's family of AI assistants, including Claude Haiku, Sonnet, and Opus.
Gemini
Google's flagship multimodal AI model family, developed by Google DeepMind.
Guardrails
Safety measures built into AI systems to prevent harmful, inappropriate, or off-topic outputs.