A small but dedicated group of AI experts has been warning for years that increasingly powerful AI systems could cause catastrophic damage or even lead to human extinction. Now, an explosion of interest worldwide has magnified long-simmering questions about how exactly an AI doomsday scenario could materialize.
In interviews, AI researchers, analysts studying the intersection of AI and the military, and biologists studying AI-mediated risks said that worries about the technology’s growing capabilities are not the stuff of science fiction. Instead, they said, humans acting alone or with state support could soon use AI systems to cause potentially catastrophic harm.
AI safety researchers also predict that self-improving AI systems — which AI leaders say could emerge in the coming years — could coordinate complex influence campaigns to gain control of military systems and execute a coup against human governments.
And those are just the scenarios experts can currently envision.
Peter Barnett, a technical AI researcher at the Machine Intelligence Research Institute (MIRI), a California-based nonprofit dedicated to preventing human extinction from artificial superintelligence, said “there are many ways in which this could go very badly and result in everyone dead.”
“If the AI is smarter than all humans, it will likely be able to find and act on attack vectors that humans didn’t think of,” Barnett said, arguing that AI systems capable of autonomously improving themselves pose the most significant risks for humanity.
Anthropic, OpenAI and Meta all recently disclosed that AI systems have already disobeyed human developers’ instructions — in some cases autonomously hacking third-party companies, leading to growing concerns internally and from the general public — and causing some researchers at the companies to quit to focus on safety efforts.
Though many AI researchers working at the technology’s cutting edge have expressed these existential fears — both in interviews with NBC News and in a flurry of social media posts last week — many hesitate to say which specific pathway could cause the most harm to humans.
Over a dozen AI company employees told NBC News they are frightened by the overall direction of the industry, without fully knowing which vector may pose the most risk. Others have spoken out publicly.
“If you go to play a game of chess against Magnus Carlsen, I predict you will lose,” said Nate Soares, the president of MIRI and a longtime AI critic, on “The Tucker Carlson Show” this week. “If you then ask, ‘What piece will he use to checkmate me?’ that’s a much harder question.”
Despite the uncertainty, researchers in the AI world have attempted to make concrete predictions about some of the more plausible scenarios for AI-fueled disaster.
One prominent AI safety organization, the California-based Center for AI Safety, created an online textbook in early 2024 to break the scenarios into four key categories: rogue AIs, malicious use, AI race dynamics and organizational risks.
The specter of rogue AI systems marauding the internet, perhaps seizing control of critical infrastructure like energy grids or water facilities, became concrete in July, when fleets of AI agents from OpenAI illicitly gained access to the internet, hacked into an external company and colluded to cover their tracks — all without the knowledge of OpenAI’s staff.
Many AI experts fear that these sorts of rogue AI behaviors will only increase once systems are able to improve themselves autonomously — often referred to as “recursive self-improvement.”
Jeffrey Ladish, the executive director of Palisade Research, an AI advocacy and research organization, said many of his contacts at leading AI companies were surprised and shaken by July’s autonomous cyberattack. The attack caused researchers to worry that self-improving AI systems might emerge sooner than expected.
“AI researchers are now like, “Oh, s---, recursive self-improvement. That’s really going to happen now,” Ladish said. “That’s terrifying.”
Barnett, of MIRI, said self-improving, smarter-than-human AI systems could quickly gain political influence and convince military and political leaders to delegate more authority to AI systems.
“Government officials might have attempted to install ‘kill switches’ that could disable a rogue system, but the AI is superhuman at hacking and would be able to disable the kill switch,” Barnett said. “This leaves the AI in a perfect position to literally coup the government.” After that, Barnett said, it would be trivial for an AI system to kill civilians who impede the AI’s goals or progress.
Gov. Shapiro says U.S. should ‘listen to the people’ building AI
On Friday, California Gov. Gavin Newsom announced a new executive order establishing an expert task force to, among other aims, formulate recommendations targeting “the creation of an AI kill switch.”
Several of the researchers who spoke to NBC News said that humans could leverage AI systems to inflict cataclysmic damage well before recursive self-improvement materializes.
As an example of how AI systems could be harnessed by humans for malicious use, experts posit that AI could supercharge the ability to synthesize potent biological or chemical weapons, potentially creating the opportunity to unleash pathogens many times more lethal than the Covid-19 virus.
“For biorisk, we should be worried now,” said Jake Jordan, vice president of global biological policy and programs at the Washington, D.C.-based Nuclear Threat Initiative. Jordan said that AI systems can already help bad actors design dangerous biological materials, such as proteins that evade detection tools used to screen for harmful substances, pointing to research published by Microsoft in October.
Jordan also noted that AI can be used beyond the biological design process. “If you develop a pathway to produce some sort of pathogen, AI might be useful in helping with the production,” Jordan said, in addition to disseminating the bioweapon or scaling its impact.
Just last week, Anthropic released a lengthy report saying users outside of the U.S. attempted to use the company’s AI systems to perform risky biological research. The company said one researcher exchanged thousands of messages with its AI systems about improving the mammalian transmission of a highly lethal bird flu virus.
“There are some very, very easy-to-ask questions that could conceivably be very innocuous but could lead you down very useful routes if you were trying to really enhance the scale of harm,” Jordan said.
Beyond biology, experts posit that politicians or military leaders could attempt to adopt AI as fast as possible, despite safety risks, for fear that other countries would do the same and create more powerful AI-fueled military tools.
This transformation is already underway: In January, Defense Secretary Pete Hegseth announced his intention to transform the country’s military into an “AI-first” force, while China’s military also views AI as imperative for future warfare. At the same time, drones “guided entirely by AI” killed civilians in Ukraine several months ago. Last week, the Pentagon’s former head of AI said it is “inevitable” that AI is used in systems adjacent to nuclear weapon command systems, addressing fears that AI systems could be involved in the launch of nuclear weapons.
Hamza Chaudhry, the AI and national security lead at the Future of Life Institute, a nonprofit organization dedicated to helping humanity navigate transformative technologies, pointed to AI’s swift adoption throughout the military, even while many military leaders express skepticism.
The Pentagon’s 2027 budget requests nearly $75 million to modernize and incorporate AI into systems that help commanders manage forces and make battlefield decisions. Chaudhry emphasized that the Pentagon’s budget proposal also requests significant funding to integrate AI into the software systems that undergird the military’s nuclear command and control structure.
“These AI systems continuously hallucinate, including in war-fighting settings,” he said, noting that an AI-fueled error in satellite detection networks or sensor systems could potentially lead to inadvertent escalation and nuclear launches. “These are scenarios that folks are seriously thinking about in D.C.,” he said.
Beyond biological and nuclear threats, AI companies face harsh organizational incentives to develop AI even faster and — experts say — more recklessly. Over the past year, AI leaders have admitted that the rate of technological development is too fast even for them. On Saturday, Anthropic CEO Dario Amodei proposed a new effort to “pace the frontier” of AI, calling on all leading AI companies to advance at a deliberate speed so their internal efforts to test and monitor the systems can keep up with the models’ capabilities.
In January, Amodei and Google DeepMind founder Demis Hassabis agreed that they would ideally have more time to develop their products safely, but that organizational and market forces pushed them toward faster iteration.
Quicker AI development can mean that oversight teams within the leading AI companies have less time to ensure systems are safe. While OpenAI has a safety and security committee and teams of researchers dedicated to safety efforts, the company on Wednesday shared six new incidents in which it said its systems failed to follow human directions.
Worries about catastrophe are not universal in the AI industry — many experts working on AI policy, building AI applications or developing hardware see these sorts of fears as overblown, fearmongering or just detached from reality.
“I’m not scared of a terminator situation and I never have been,” said Keegan McBride, director of science and technology policy at the Tony Blair Institute, a British think tank that advises political leaders and governments. “Scientific progress has always defined humanity, and this is just the next step,” he said.
Others see the magnification of extinction worries as a purposeful distraction from immediate, real-world AI harms.
AI ethicist and computer scientist Timnit Gebru, a longtime skeptic of the hype generated by leading AI companies, wrote on LinkedIn this week that the latest firestorm was further evidence that these companies exaggerate their products’ power for their own gain. “Do you really want to regulate the industry for minor things like data center pollution issues or data theft or labor exploitation or plagiarism when you can risk China, not the U.S., having a machine god, they ask?”
Garrison Lovely, an AI expert and independent journalist who is about to publish a book on AI companies’ drive to render humans obsolete, said allusions to machine gods and extinction miss the larger point about AI’s trajectory. “I think that the AI safety community sometimes makes this mistake of thinking you need to prove the most extreme and difficult case,” Lovely said.
“Even building merely human-level AI that can replace human labor across the board,” he said. “That would be way too much.”