{"slug": "why-so-many-ai-researchers-think-the-machines-could-kill-everyone", "title": "Why So Many AI Researchers Think the Machines Could Kill Everyone", "summary": "AI researcher Jacob Coxon resigned from Anthropic this week, warning that AI firms are \"racing straight to self-improving superintelligence and gambling with our lives,\" while a senior Anthropic safety leader said on X that AI could kill all humans with a probability he personally puts at more than 10% within the next decade. Coxon's departure follows that of Rishub Jain, who quit Google DeepMind in June over fears that recursive self-improvement—using AI's coding skills to build its own successors—is removing humans from the loop, a concern echoed by MIRA computer scientist Nate Soares, who says the vision is \"starting to feel real.\" No frontier lab claims to have achieved a fully autonomous improvement cycle, but the concept has drawn warnings from Anthropic and funding for startups such as Recursive Intelligence.", "body_md": "Earlier this year, Rishub Jain left his position as an [artificial intelligence](https://www.wired.com/tag/artificial-intelligence/) researcher at [Google DeepMind](https://www.wired.com/tag/deepmind/) after a revelation.\n\nAs he worked on new models, he came to believe that he and everyone else on AI’s frontier were ceding control. By using AI’s coding skills to accelerate work on the next generation of models, he was removing himself from the equation. AI labs hope to evolve this approach to the point that AI will improve itself indefinitely, a process known as recursive self-improvement.\n\nJain believed that keeping humans in the picture might be crucial to maintaining control over the technology—and avoiding dire consequences. “AI progress is increasing,” he tells WIRED. “And as AI becomes more capable, it poses more risks.” The idea that he may not have proper visibility into how an AI model was building its successor made him so uneasy that, in June, he quit.\n\nJain is one of a growing number of AI researchers speaking out over those fears.\n\nThe panic has intensified in recent weeks. Genuinely stunning advances in AI capabilities—an OpenAI model solved a [centuries-old math problem](https://www.wired.com/story/openai-navier-stokes-math-discovery-academics/) in a matter of hours—have come amid a rash of security incidents that saw swarms of agents [break free from containment](https://www.wired.com/story/openai-didnt-notice-its-ai-agents-using-a-message-board-to-plan-their-hacking-spree/) to hack into other systems.\n\nThose concerns reached a fever pitch this week after researcher Jacob Coxon announced his resignation from Anthropic [while warning](https://www.wired.com/story/anthropic-researcher-quits-jacob-coxon-ai-fears-humanity/) that AI firms are “racing straight to self-improving superintelligence and gambling with our lives.” A senior Anthropic leader—who works on AI safety—[piped up](https://x.com/EvanHub/status/2097497037956891126) with a similarly blunt assessment: “We really do earnestly believe AI could kill all humans! I personally think it is >10% within the next decade.”\n\n“I do think that the vision of recursive self-improvement is spooking people,” says Nate Soares, a computer scientist at MIRA, a research nonprofit, and the coauthor of [*If Anybody Builds It, Everybody Dies*](https://www.wired.com/story/the-doomers-who-insist-ai-will-kill-us-all/), which argues that superhuman AI would lead to human extinction. “It’s starting to feel real.”\n\nA key component of recursive self-improvement is the idea of a feedback loop that automates the development process so that AI becomes increasingly powerful. No frontier AI lab claims to have achieved this sort of fully autonomous cycle of improvement; it remains theoretical for now. But it has inspired the launch of some well-funded startups such as [Recursive Intelligence](https://www.recursive.com/), as well as [warnings](https://www.anthropic.com/institute/recursive-self-improvement) from big firms about unintended outcomes straight out of “The Sorcerer’s Apprentice.”\n\nSoares, who pioneered work on alignment, a technical field that involves trying to match AI with human values, says it’s also becoming more evident that there is no practical way to guarantee that AI will behave itself.\n\n“I think a lot of people had this fantasy that [alignment] was going to get easier as these things got smarter, and now it’s getting harder. And they’re like, ‘Oh shit,’” he says.\n\nSoares says he regularly talks to people inside the big AI labs who are worried about the potential consequences of the research they’re doing. “I tend to recommend they quit, and they say it wouldn’t do anything,” he says. “And then Jacob quits, and we see who was right.”\n\nDaniel Kokotajlo, the author of [AI 2027](https://ai-2027.com/), an influential project warning about the dangers of increasingly powerful AI, shares fears about recursive self-improvement. The version of this work currently being done often involves dispatching thousands of agents to collaborate on a problem, something that further abstracts away oversight and control because of the vast complexity involved.\n\nMany doomsayers seem to agree that the incentives for big AI companies are hardly aligned with good outcomes, especially as OpenAI and Anthropic barrel toward their respective IPOs. “At Anthropic, the stakes are well understood, but they are locked in a race to get there first,” Coxon wrote on X.\n\nKokatajlo points out that the drumbeat of concern was growing well before Coxon’s viral resignation, the numerous hacking incidents, and the math breakthrough. Anthropic executives have said since the company’s founding that AI could represent an existential threat. In July over a thousand top AI engineers [signed an open letter](https://www.pacingthefrontier.com/) calling for a coordinated slowdown in the development of advanced AI. He attributes the recent flurry of concern to the specter of recursive self-improvement more than anything else.\n\nBut it also comes at a time when people are concerned about massive data center build-outs and potential job losses from AI. Trust in AI companies—and AI researchers themselves—may be reaching an all-time low.\n\n“People are waking up and saying ‘the companies are actually trying to build superintelligence … what? That’s insane,’” Kokatajlo says.\n\nJust how risky it is to carry on building AI is hard to quantify. But when pushed to explain exactly *how* AI might go about eliminating the species that created it, Soares suggests it could happen in a number of ways. It could involve manipulating humans to trigger a catastrophe, or controlling an army of killer robots.\n\nOne of the more easy-to-imagine scenarios could involve AI that is hooked up to a biolab, Soares suggests. “We could say we’ll turn it off, but it could say, ‘Unfortunately, I have your off switch, which is this super virus.’” (Coxon also floated the idea of a new virus in an interview with WIRED, while Anthropic [said](https://www.nytimes.com/2026/09/10/us/politics/anthropic-ai-biological-weapons.html) Thursday that it had cut off access to several outside researchers over fears about bioweapons.)\n\nAI hardly needs to wipe out humanity in order to be harmful, though. Many experts predict that more powerful models will lead to a coming wave of [AI-assisted cyberattacks. The technology is now widely used for disinformation campaigns, and military adoption of AI is accelerating rapidly.](https://www.wired.com/story/security-news-this-week-the-cybersecurity-apocalypse-is-coming-in-months-ai-giants-warn/)\n\nStill, not everyone sees doom as inevitable. Jain, the ex-Google DeepMind researcher, recently launched Sampura Research, a company working to develop techniques for aligning models that involve keeping humans in the loop, even if AI does the lion’s share of assessing whether behavior is good or bad. He notes that there is now significant funding for AI safety startups like his.\n\nJain seems hopeful that AI can be tamed yet. “You can ask an AI, ‘Is this task safe?’ and it judges that, but we think that combining both AI and humans to do that task will lead to even better performance,” he says.", "url": "https://wpnews.pro/news/why-so-many-ai-researchers-think-the-machines-could-kill-everyone", "canonical_source": "https://www.wired.com/story/why-so-many-ai-researchers-think-the-machines-could-kill-everyone/", "published_at": "2026-09-11 09:00:00+00:00", "updated_at": "2026-09-11 09:32:11.493036+00:00", "lang": "en", "topics": ["ai-safety", "artificial-intelligence", "ai-research", "ai-policy", "ai-startups"], "entities": ["Rishub Jain", "Google DeepMind", "Jacob Coxon", "Anthropic", "Nate Soares", "MIRA", "Recursive Intelligence", "Daniel Kokotajlo"], "alternates": {"html": "https://wpnews.pro/news/why-so-many-ai-researchers-think-the-machines-could-kill-everyone", "markdown": "https://wpnews.pro/news/why-so-many-ai-researchers-think-the-machines-could-kill-everyone.md", "text": "https://wpnews.pro/news/why-so-many-ai-researchers-think-the-machines-could-kill-everyone.txt", "jsonld": "https://wpnews.pro/news/why-so-many-ai-researchers-think-the-machines-could-kill-everyone.jsonld"}}