I think one form the first big AI hack could take is a prompt injection worm. Let's piece this together.
So basically, one day we wake up and terabytes of sensitive data has been uploaded to the attackers and/or dropped publicly online for embarrassment purposes. This might include credentials, customer data, whatever.
Another variation of this attack could be a much smaller scope, but more targeted, where the credentials are actually used quietly versus blasted out all at once. The issue with doing the first version is that it will be so loud that everyone will check and start rotating credentials. Whereas if someone does the second version, it will take a lot longer for them to figure out they were compromised.
The most interesting and concerning part of this to me is that this is a game of the strength of prompt injection defenses versus the rapidly increasing intelligence of unrestricted open source models. And I don't like the odds for us in this fight.
There have already been lots of other types of AI-harness-based attacks of the more traditional form, and those will surely continue as well, but I see the combination of prompt injection with the massive number of parsers and integrations as one that will hit soon.
Without hyperbole, I think what they announced represents both the greatest boon for business and the biggest problem for security that we've seen injected in a single day in many decades.
[AI Agents + API Access + Prompt Injection], November 2023
So, what to do about it?
You have to know where your parsers are. In other words, you have to know where you have AI touching your tech stacks and workflows. You have to look at all your integrations, continuously, and have threat models for them based on what they have access to.
One of the biggest security problems we'll face around AI will be semi-autonomous agents roaming the internet with too much authority. There are two main issues: parsing everything without consideration, and being connected to internal functionality while doing so.
[AI Canaries], June 2023 Then you have to stack your defensive layers for prevention, and perhaps even more importantly, be ready to respond if something happens.
If I'm right, this is the quiet before the storm hits.