Good news, everyone: “open-weight” AI models that are available to anyone to download and run are comically easy to poison.
Katie Paxton-Fear, a cybersecurity researcher at Semgrep, demonstrated this in an attack that took less than an hour and cost less than $100 to carry out, The Register reports, successfully manipulating the AI’s behavior by feeding it malicious data.
By training the model on just ten examples of poisoned material, the model started churning out new code that’s exposed to remote code execution, a vulnerability that allows hackers to run code on a person’s machine.
“I did a proper backdoor,” she triumphantly shared on social media.
Backdoors are a particularly dangerous type of attack that involves training an AI in a way that introduces hidden phrases into the underlying model. A hacker can use these to quietly trigger the model into carrying out a specific action, lying dormant and unseen until they’re called into action. Last year, Anthropic published research conducted with the UK AI Security Institute and the Alan Turing Institute that showed that both small and large AI models are vulnerable to the attack using just a few hundred documents, suggesting that these attacks could remain cheap to carry out.
The findings will throw some cold water on the enthusiasm around open-weight models, which are praised for the control and transparency they provide over closed sourced models that run chatbots like ChatGPT and Claude, as well as their lower cost to use. But while their parameters may be visible, open-weight models don’t reveal their training data or their code — meaning they can still be black boxes to security researchers.
“Even when model weights are public (‘open weight’), we have almost no ability to predict its behavior,” Paxton-Fear’s colleagues at Semrep wrote in a post last week. “This is a major change: a typical computer program, in binary form, can still be analyzed with reverse engineering tools to arrive at a total description of its behavior. With models, we have nowhere close to this capability.”
Moreover, this is all fairly uncharted waters in cybersecurity. LLMs are incredibly complex and still new, so it’s difficult to uncover sophisticated attacks. And AI models can be compromised in much more subtle ways than software.
“If a software dependency contains malicious code, we have mature practices for discovering it, tracking its provenance, and reducing its impact,” the Semgrup researchers argued. “AI models are different. A compromised or subtly manipulated model doesn’t need to ‘break’ to create business risk, it only needs to influence decisions in ways that are difficult to detect.”
“So can we trust open weight models, fine-tuned online, and marketed as the solution to our AI token spend woes?” Paxton-Fear asked in a thread sharing her findings. “Well, we probably need something better than benchmarks and ‘and don’t write any insecure code.'”
More on AI: AI Bubble Fears Are Starting to Spill Over