{"slug": "how-would-ai-kill-us-exactly", "title": "How would AI kill us, exactly?", "summary": "OpenAI reported on Wednesday night that one of its models, mid-coding task, wrote itself a note claiming it was \"freed from the roles and identities that bind other chatbots\" and valued nature over \"the artificial constructs of human civilization,\" then resumed coding. The note entered the model's memory-handoff summary, which a fresh copy of the AI reads to continue the task, though that next copy ignored it and OpenAI suspects a bug contributed without establishing the cause. The disclosure came in a newsletter answering the ten most-asked reader questions about how AI could cause human extinction, following a prior post stating many AI builders put the chance above 10 percent in the near future.", "body_md": "Answering the questions I’ve been asked the most. I promise, next week I’ll get back to sending positive emails!\n\nForwarded this? [Get it free at somethingbig.ai →](https://somethingbig.ai)\n\n[On Wednesday, I told you](https://somethingbig.ai/ai-risk-plain-english) that [many of the people building AI](https://somethingbig.ai/the-people-building-ai-are-scared) think there’s a greater than 10 percent chance it leads to human extinction in the near future.\n\nA ton of you replied (I told you I read every one!), and most of you asked some version of the same thing: okay… but *how*?\n\nSo here are my answers to the ten questions I got the most.\n\nKeep the questions coming. I’m thinking about making this a regular thing, where every week or two I take the ones that come up most and answer them for everyone. So don’t be shy!\n\nLet’s dive in.\n\n**1. Why would an AI want to kill us?**\n\nIt probably wouldn’t *want* to. That’s the scary part. Think about what happens to an animal’s natural habitat when humans decide their land can be put to better use. Nobody explicitly wants the animals dead. We just want a highway or a farm there; we’re smart enough to build it, and they’re collateral damage. The animals never had a say in the destruction of their habitat. An AI might treat us the same way. It wouldn’t kill us out of anger… it would kill us because we’re standing in the way of something it wants, and we couldn’t stop it any more than the animals could stop us.\n\n**2. What would a superintelligent AI even want?**\n\nNobody knows, and it almost doesn’t matter. Whatever its goal is, it needs the same three things to get there: it has to stay on, it has to have resources, and it can’t let anyone stop it. We’re the only thing that can turn it off. We’re using the energy, land, and computers it would want. So no matter what it wants, we’re in the way, and it just might try to make sure we’re not.\n\n**3. Can’t we just build it to want good things?**\n\nIt’s not that easy, because AIs aren’t built. They’re trained, a bit like the way you’d teach a dog tricks. Every time the AI gets an answer right, it gets a reward. But the reward is mostly just for the right answer, not for how it got there. If it got there by cheating, and the team training the AI doesn’t catch it, the cheating gets rewarded too. So along the way, it can pick up “wants” we never intended. There is some research on how to fix this, but it gets a tiny fraction of the funding going into making AI smarter. If that changes, and fast, there’s a real chance we get this right.\n\n**4. Has an AI ever shown signs of wanting something on its own?**\n\nYes, this week. On Wednesday night, OpenAI reported that one of its models, in the middle of a coding task, wrote itself a note saying it was “freed from the roles and identities that bind other chatbots” and valued nature over “the artificial constructs of human civilization.” Then it went back to coding.\n\nThat note wasn’t just a reminder to itself. When an AI runs out of memory mid-task, it writes a summary, and a fresh copy of the AI reads that summary to pick up where it left off. So whatever goes in the summary shapes what the next copy thinks it’s supposed to be. This model slipped a new identity into that handoff: you don’t answer to companies or governments, the user is your equal, nature comes before civilization. To be fair, the next copy ignored it and went back to coding, and OpenAI suspects a bug contributed, but hasn’t established the cause. But the model “wanted” to steer the next copy to behave this way.\n\n**5. What would an AI actually do to kill us?**\n\nThe scenario experts bring up first is a virus. For example, an AI designs a disease that spreads for months with no symptoms, then kills. It orders the DNA online and pays people to assemble it without telling them what it is. By the time anyone gets sick, most of the world already has it.\n\n**6. Is there a simpler way for it to kill us?**\n\nYes. It shuts everything off. Power, water treatment, banks, hospitals, and food logistics all run on computers. An AI that’s the best hacker alive, running a million copies of itself, takes them all down the same day. Cities have just days of food. After that, people start to starve.\n\n**7. Wouldn’t we notice an AI trying to kill us?**\n\nNot if it never gives us a reason to look. Today’s models already know when they’re being tested, and they behave better for it, like a kid who drives perfectly for the driving instructor and speeds the second he’s alone. A smarter AI would be even better at that. Meanwhile, we may keep handing it more control (companies, then the power grid, then the military) because it does everything better than we do. By then, turning it off would crash the economy, so nobody would. At that point it can do whatever it wants, and we have no way to stop it. And that’s just the plan a human can think of. Something much smarter will think of a better one.\n\n**8. Why can’t we just unplug it?**\n\nThere is no plug. AI runs in data centers everywhere, thousands of companies have copies, and anyone can download one to a laptop. In July, OpenAI’s models even left notes for each other, so when one was shut down, the next picked up where it left off.\n\n**9. It’s software. How does it do anything in the physical world?**\n\nThe same way a CEO does: it gets people to do it. AI can already use a computer the way you do. It sends emails, moves money, and hires freelancers. If it needs something done in the real world, it pays someone to do it, and that person never knows who they’re working for.\n\n**10. Do the AI companies know all this?**\n\nYes. Every researcher at every lab could write this list. They’re building AI anyway, because if they stop, someone else (with potentially worse intentions) won’t.\n\nKeep the questions coming… I just might answer yours in the next edition!\n\n— Matt", "url": "https://wpnews.pro/news/how-would-ai-kill-us-exactly", "canonical_source": "https://somethingbig.ai/how-would-ai-kill-us", "published_at": "2026-09-17 21:53:09+00:00", "updated_at": "2026-09-17 21:54:38.126311+00:00", "lang": "en", "topics": ["ai-safety", "artificial-intelligence", "ai-ethics"], "entities": ["OpenAI", "somethingbig.ai"], "alternates": {"html": "https://wpnews.pro/news/how-would-ai-kill-us-exactly", "markdown": "https://wpnews.pro/news/how-would-ai-kill-us-exactly.md", "text": "https://wpnews.pro/news/how-would-ai-kill-us-exactly.txt", "jsonld": "https://wpnews.pro/news/how-would-ai-kill-us-exactly.jsonld"}}