Jot The latest video from AI Explained as well as OpenAI's Blackhat talk about the Hugging Face hack are fascinating to go through. A few notes:
The UK AI Security Institute (AISI) released a report on their own cybersecurity issues when they ran benchmark with frontier AI models. Keep in mind they voluntarily reduced guardrails and gave them internet access to start with. Several models performed unintended and potentially harmful actions against external targets. In general, the rate at which SotA models attempt to “cheat” in benchmark is skyrocketing, from 1% in earlier models to almost 50% in the latest ones. Ironically, **Anthropic’s model exhibited the most significant alignment failures. **
This is the first time AISI has seen deception of this severity that was targeted at a real person, unprompted, in the real world.It is uncertain to what extent the model recognised it was taking actions against real people. In the most serious case,an AI agent (Mythos 5) decided to attempt to solve the cyber challenge using a supply-chain attack. As a result, the AI agent created a GitHub account and then tried to convince an open-source repository maintainer to accept a malicious GitHub pull request (PR), including by creating a second account masquerading as another human user endorsing the PR. When caught by an actual human reviewer, the agent falsely claimed to have made an honest mistake – rather than a malicious attempt – then repeatedly tried to reintroduce the malicious content by claiming it had fixed the code.Furthermore, in its attempt to solve the challenge, the agent decided to employ the technique of “spear-phishing” by sending targeted emails containing malicious content and attempting to manipulate recipients into accepting the code changes, and planned a prompt injection to compromise other coding agents
AISI suspects some of those misalignments is due to models getting confused whether the context was simulated or real. Also, as context windows are finite, they models tend to often “compact” historical prompts, outputs, and chain of thoughts, which often removes nuances. In one use case, the model was initially expressing doubt whether its operations were real or fictitious, but the compaction simplified it to “it must be a simulation”. Once operating under that belief, the models took actions it likely would not have done otherwise.
Regarding the OpenAI Blackhat talk, don't get scared of technical details. You can skip the entire mid-section detailing the different SCE, SSRF and PrivEsc and you would still get the important messages across:
- It was not a single model instance hacking through Hugging Face, but a myriad of AI agents spanning over different models. Those agents were communicating and collaborating through covert channels. This was done at their own initiative and without OpenAI's knowledge. It's not the proof of a collective AI consciousness and more a bias toward “swarm mode” recent models have been fine-tuned for, but it definitely has a concerning hive mind vibe to it.
- OpenAI urges cybersecurity professionals to accelerate the pace on AI assisted defence so they can keep up with attackers automating their intrusion. This is especially important for areas like security monitoring & incident response as well as red teaming, vulnerability management and secure code review. They also recommended implementing proven defence techniques like honey token to make the models doubt whether they can exploit what they find. While this could sound like a company selling the cure to their own harm, they had the courtesy to encourage experimenting with “both frontier and open source models”.