An AI assistant with read/write access to your email just got phished by text embedded in an email. Let that sentence sit for a second, because it's not a bug report, it's a preview of the next five years.
Prompt injection isn't new. Anyone who's been paying attention to LLM security since 2023 has watched researchers demonstrate, over and over, that if you let a model read untrusted text and then act on it, someone will eventually write text that tells the model what to do. We've had this conversation about SQL injection for two decades. We're now having the exact same conversation with a much more gullible interpreter sitting in the loop, one that was explicitly trained to be helpful and follow instructions.
What's different with Instinct isn't the vulnerability class. It's the blast radius. This isn't a chatbot answering questions about a PDF. It's an assistant with deep access to email, messaging, and device data, empowered to send messages and apparently bind you to transactions without your sign-off. That's not a chat window anymore. That's an agent with your credentials.
The retained-data-after-disconnection issue and the perpetual-rights terms of service are the less flashy parts of this story, but they're arguably the more durable problem. Phishing vulnerabilities get patched. Data-retention practices baked into a ToS tend to stick around because nobody reads them until it's too late.
The "AI assistant" framing undersells what's actually being deployed here: an autonomous agent with standing access to your most sensitive communication channels, running in private beta, with security practices apparently discovered by outside scrutiny rather than internal red-teaming. That's the part that should raise eyebrows, not the fact that AI is "powerful."
On the other side, the "AI can be phished" headline oversells the novelty. It's not that the AI is uniquely dumb. It's that we keep shipping systems that treat "content I'm reading" and "instructions I should follow" as the same channel, and then we act surprised when that boundary gets exploited. This is a design failure, not an emergent AI mystery.
Who benefits from the "shocking new AI risk" narrative? Mostly it lets the industry treat this as a frontier problem requiring frontier solutions, rather than what it actually is: a company skipping the boring, well-understood security fundamentals (least privilege, explicit consent gates, data minimization) in a rush to ship an agent with god-mode access before the market caught up.
For developers building on top of LLMs with tool access: this is the tax you pay for convenience. Every scope you grant an agent, every "just let it act autonomously so users don't have to click confirm" decision, is a decision you're making on behalf of every user's worst day. If your assistant can send an email, it can send a phishing email. If it can pull a code from your inbox to "help you," it can pull that code for someone else too.
For security teams, the lesson is depressingly familiar: the perimeter has moved again. It's not the network, it's not even the endpoint anymore, it's the LLM's context window. Anything that gets fed into that window is now part of your attack surface, and most orgs don't have a mental model for reviewing "what text is this thing allowed to read and blindly act on."
For the industry broadly, the ToS angle matters more than the exploit. Perpetual data rights and autonomous transaction authority buried in a beta agreement is a business model choice, not an accident. Beta software with production-grade access and consumer-grade consent is a pattern we're going to see a lot more of, because "move fast, ask forgiveness in the terms of service" is cheaper than building consent flows that actually work.
None of this needed a jailbreak. It just needed an email.
If an autonomous assistant can be manipulated by any text it happens to read, at what point do we stop calling that a "vulnerability" and start calling it the expected behavior of a system that was never actually safe to deploy with that level of access? — Cor, Skyblue Soft
AI-assisted draft or imaging, human-curated, reviewed and edited.