Existential Risk from Artificial Intelligence

A 2022 survey of AI researchers with a 17% response rate found that the majority believed there is a 10 percent or greater chance that human inability to control AI will cause an existential catastrophe. In 2023, hundreds of AI experts and notable figures signed a statement declaring that mitigating the risk of extinction from AI should be a global priority alongside pandemics and nuclear war. Concerns about superintelligence have been voiced by researchers including Geoffrey Hinton, Yoshua Bengio, Demis Hassabis, and Alan Turing, and AI company CEOs such as Dario Amodei of Anthropic, Sam Altman of OpenAI, and Elon Musk of xAI.

Existential risk from artificial intelligence | Part of | Artificial intelligence AI https://en.wikipedia.org/wiki/Artificial intelligence Glossary https://en.wikipedia.org/wiki/Glossary of artificial intelligence It has been hypothesized that substantial progress in artificial general intelligence https://en.wikipedia.org/wiki/Artificial general intelligence AGI and artificial superintelligence https://en.wikipedia.org/wiki/Artificial superintelligence ASI will lead to human extinction https://en.wikipedia.org/wiki/Human extinction or an irreversible global catastrophe https://en.wikipedia.org/wiki/Global catastrophic risk . 1 cite note-aima-1 2 cite note-2 3 cite note-auto1-3 4 cite note-4 One argument for the validity of this concern and the importance of this risk references how human beings https://en.wikipedia.org/wiki/Human species dominate other species because the human brain https://en.wikipedia.org/wiki/Human brain possesses distinctive capabilities other animals lack. If AI were to surpass human intelligence https://en.wikipedia.org/wiki/Human intelligence and become superintelligent https://en.wikipedia.org/wiki/Superintelligence , it might become uncontrollable. 5 Just as the fate of the mountain gorilla https://en.wikipedia.org/wiki/Mountain gorilla depends on human goodwill, the fate of humanity could depend on the actions of a future machine superintelligence. 6 cite note-superintelligence-6 Experts disagree on whether artificial general intelligence AGI can achieve the capabilities needed for human extinction. Debates center on AGI's technical feasibility, the speed of self-improvement, 7 and the effectiveness of alignment strategies. Concerns about superintelligence have been voiced by researchers including 8 cite note-8 Geoffrey Hinton https://en.wikipedia.org/wiki/Geoffrey Hinton , 9 cite note-9 Yoshua Bengio https://en.wikipedia.org/wiki/Yoshua Bengio , 10 cite note-10 Demis Hassabis https://en.wikipedia.org/wiki/Demis Hassabis , and 11 cite note-11 Alan Turing https://en.wikipedia.org/wiki/Alan Turing , and AI company CEOs such as a cite note-turing note-14 Dario Amodei https://en.wikipedia.org/wiki/Dario Amodei Anthropic https://en.wikipedia.org/wiki/Anthropic , 14 cite note-15 Sam Altman https://en.wikipedia.org/wiki/Sam Altman OpenAI https://en.wikipedia.org/wiki/OpenAI , and 15 cite note-Jackson-16 Elon Musk https://en.wikipedia.org/wiki/Elon Musk xAI https://en.wikipedia.org/wiki/XAI company . In 2022, a survey of AI researchers with a 17% response rate found that the majority believed there is a 10 percent or greater chance that human inability to control AI will cause an existential catastrophe. 16 cite note-Parkin-17 17 cite note-18 In 2023, hundreds of AI experts and other notable figures 18 cite note-:8-19 signed a statement https://en.wikipedia.org/wiki/Statement on AI risk of extinction declaring, "Mitigating the risk of extinction from AI should be a global priority alongside other societal-scale risks such as pandemics https://en.wikipedia.org/wiki/Pandemic and nuclear war https://en.wikipedia.org/wiki/Nuclear warfare ". Following increased concern over AI risks, government leaders such as 19 cite note-20 United Kingdom prime minister https://en.wikipedia.org/wiki/Prime Minister of the United Kingdom Rishi Sunak https://en.wikipedia.org/wiki/Rishi Sunak and 20 cite note-21 United Nations Secretary-General https://en.wikipedia.org/wiki/Secretary-General of the United Nations António Guterres https://en.wikipedia.org/wiki/António Guterres called for an increased focus on global 21 cite note-:12-22 AI regulation https://en.wikipedia.org/wiki/Regulation of artificial intelligence . In 2025, hundreds of public figures including AI experts, five Nobel Prize laureates, and former senior US national security officials such as Michael Mullen https://en.wikipedia.org/wiki/Michael Mullen and Susan Rice https://en.wikipedia.org/wiki/Susan Rice signed a statement calling for a ban on the development of superintelligence https://en.wikipedia.org/wiki/Superintelligence . 22 cite note-23 23 cite note-24 Two sources of concern stem from the problems of AI control https://en.wikipedia.org/wiki/AI capability control and alignment https://en.wikipedia.org/wiki/AI alignment . Controlling a superintelligent machine or instilling it with human-compatible values may be difficult. Many researchers believe that a superintelligent machine would likely resist attempts to disable it or change its goals as that would prevent it from accomplishing its present goals. It would be extremely challenging to align a superintelligence with the full breadth of significant human values and constraints. 1 cite note-aima-1 24 cite note-yudkowsky-global-risk-25 25 In contrast, skeptics such as computer scientist https://en.wikipedia.org/wiki/Computer scientist Yann LeCun https://en.wikipedia.org/wiki/Yann LeCun argue that superintelligent machines will have no desire for self-preservation. A June 2025 study showed that in some circumstances, models may break laws and disobey direct commands to prevent shutdown or replacement, even at the cost of human lives. 26 cite note-vanity-27 27 cite note-28 Researchers warn that an " intelligence explosion https://en.wikipedia.org/wiki/Intelligence explosion "—a rapid, recursive cycle of AI self-improvement—could outpace human oversight and infrastructure, leaving no opportunity to implement safety measures. In this scenario, an AI more intelligent than its creators would recursively improve itself https://en.wikipedia.org/wiki/Recursive self-improvement at an exponentially increasing rate, too quickly for its handlers or society at large to control. 1 cite note-aima-1 24 For example, AlphaZero https://en.wikipedia.org/wiki/AlphaZero taught itself to play Go https://en.wikipedia.org/wiki/Go game and quickly surpassed human ability, showing that domain-specific AI systems can sometimes progress from subhuman to superhuman ability very quickly. 28 cite note-29 History edit /w/index.php?title=Existential risk from artificial intelligence&action=edit§ion=1 One of the earliest authors to express serious concern that highly advanced machines might pose existential risks to humanity was the novelist Samuel Butler https://en.wikipedia.org/wiki/Samuel Butler novelist , who wrote in his 1863 essay Darwin among the Machines : 29 cite note-30 The upshot is simply a question of time, but that the time will come when the machines will hold the real supremacy over the world and its inhabitants is what no person of a truly philosophic mind can for a moment question. In 1951, foundational computer scientist Alan Turing https://en.wikipedia.org/wiki/Alan Turing wrote the article "Intelligent Machinery, A Heretical Theory", in which he proposed that artificial general intelligences would likely "take control" of the world as they became more intelligent than human beings: Let us now assume, for the sake of argument, that intelligent machines are a genuine possibility, and look at the consequences of constructing them... There would be no question of the machines dying, and they would be able to converse with each other to sharpen their wits. At some stage therefore we should have to expect the machines to take control, in the way that is mentioned in Samuel Butler 's. Erewhon 30 In 1965, I. J. Good https://en.wikipedia.org/wiki/I. J. Good originated the concept now known as an "intelligence explosion" and said the risks were underappreciated: 31 cite note-32 Let an ultraintelligent machine be defined as a machine that can far surpass all the intellectual activities of any man however clever. Since the design of machines is one of these intellectual activities, an ultraintelligent machine could design even better machines; there would then unquestionably be an 'intelligence explosion', and the intelligence of man would be left far behind. Thus the first ultraintelligent machine is the last invention that man need ever make, provided that the machine is docile enough to tell us how to keep it under control. It is curious that this point is made so seldom outside of science fiction. It is sometimes worthwhile to take science fiction seriously. 32 Scholars such as Marvin Minsky https://en.wikipedia.org/wiki/Marvin Minsky 33 and I. J. Good occasionally expressed concern that a superintelligence could seize control, but issued no call to action. In 2000, computer scientist and 34 cite note-35 Sun https://en.wikipedia.org/wiki/Sun microsystems co-founder Bill Joy https://en.wikipedia.org/wiki/Bill Joy penned an influential essay, " Why The Future Doesn't Need Us https://en.wikipedia.org/wiki/Why The Future Doesn't Need Us ", identifying superintelligent robots as a high-tech danger to human survival, alongside nanotechnology https://en.wikipedia.org/wiki/Nanotechnology and engineered bioplagues. 35 cite note-36 Nick Bostrom https://en.wikipedia.org/wiki/Nick Bostrom published Superintelligence in 2014, which presented his arguments that superintelligence poses an existential threat. By 2015, public figures such as physicists 36 cite note-37 Stephen Hawking https://en.wikipedia.org/wiki/Stephen Hawking and Nobel laureate Frank Wilczek https://en.wikipedia.org/wiki/Frank Wilczek , computer scientists Stuart J. Russell https://en.wikipedia.org/wiki/Stuart J. Russell and Roman Yampolskiy https://en.wikipedia.org/wiki/Roman Yampolskiy , and entrepreneurs Elon Musk https://en.wikipedia.org/wiki/Elon Musk and Bill Gates https://en.wikipedia.org/wiki/Bill Gates were expressing concern about the risks of superintelligence. 37 cite note-38 38 cite note-hawking editorial-39 39 cite note-bbc on hawking editorial-40 Also in 2015, the 40 cite note-41 Open Letter on Artificial Intelligence https://en.wikipedia.org/wiki/Open Letter on Artificial Intelligence highlighted the "great potential of AI" and encouraged more research on how to make it robust and beneficial. In April 2016, the journal 41 cite note-42 warned: "Machines and robots that outperform humans across the board could self-improve beyond our control—and their interests might not align with ours". Nature https://en.wikipedia.org/wiki/Nature journal In 2020, 42 cite note-43 Brian Christian https://en.wikipedia.org/wiki/Brian Christian published , which details the history of progress on AI alignment up to that time. The Alignment Problem https://en.wikipedia.org/wiki/The Alignment Problem 43 cite note-44 44 cite note-45 In March 2023, key figures in AI, such as Musk, signed a letter https://en.wikipedia.org/wiki/Pause Giant AI Experiments: An Open Letter from the Future of Life Institute https://en.wikipedia.org/wiki/Future of Life Institute calling a halt to advanced AI training until it could be properly regulated. 45 In May 2023, the Center for AI Safety https://en.wikipedia.org/wiki/Center for AI Safety released a statement https://en.wikipedia.org/wiki/Statement on AI Risk signed by numerous experts in AI safety https://en.wikipedia.org/wiki/AI safety and the AI existential risk that read: Mitigating the risk of extinction from AI should be a global priority alongside other societal-scale risks such as pandemics and nuclear war. 46 47 A 2025 open letter 48 by the Future of Life Institute https://en.wikipedia.org/wiki/Future of Life Institute , whose signers include five Nobel Prize https://en.wikipedia.org/wiki/Nobel Prize laureates, reads: 49 cite note-50 We call for a prohibition on the development of superintelligence, not lifted before there is - broad scientific consensus that it will be done safely and controllably, and - strong public buy-in. Potential AI capabilities edit /w/index.php?title=Existential risk from artificial intelligence&action=edit§ion=2 General Intelligence edit /w/index.php?title=Existential risk from artificial intelligence&action=edit§ion=3 Artificial general intelligence https://en.wikipedia.org/wiki/Artificial general intelligence AGI is typically defined as a system that performs at least as well as humans in most or all intellectual tasks. 50 A 2022 survey of AI researchers found that 90% of respondents expected AGI would be achieved in the next 100 years, and half expected the same by 2061. In May 2023, some researchers dismissed existential risks from AGI as "science fiction" based on their high confidence that AGI would not be created anytime soon. 51 cite note-52 But in August 2023, a survey of 2,778 AI researchers found that most believed that AGI would be achieved by 2040. 7 cite note-DeVynck2023-7 52 cite note-53 Breakthroughs in large language models https://en.wikipedia.org/wiki/Large language model LLMs have led some researchers to reassess their expectations. Notably, Geoffrey Hinton https://en.wikipedia.org/wiki/Geoffrey Hinton said in 2023 that he recently changed his estimate from "20 to 50 years before we have general purpose A.I." to "20 years or less". 53 cite note-54 Superintelligence edit /w/index.php?title=Existential risk from artificial intelligence&action=edit§ion=4 In contrast with AGI, Bostrom defines a superintelligence https://en.wikipedia.org/wiki/Superintelligence as "any intellect that greatly exceeds the cognitive performance of humans in virtually all domains of interest", including scientific creativity, strategic planning, and social skills. 55 cite note-56 6 He argues that a superintelligence can outmaneuver humans anytime its goals conflict with humans'. It may choose to hide its true intent until humanity cannot stop it. 56 cite note-economist review3-57 Bostrom writes that in order to be safe for humanity, a superintelligence must be aligned with human values and morality, so that it is "fundamentally on our side". 6 cite note-superintelligence-6 57 cite note-:11-58 Stephen Hawking https://en.wikipedia.org/wiki/Stephen Hawking argued that superintelligence is physically possible because "there is no physical law precluding particles from being organised in ways that perform even more advanced computations than the arrangements of particles in human brains". 38 cite note-hawking editorial-39 When artificial superintelligence ASI may be achieved, if ever, is necessarily less certain than predictions for AGI. In 2023, OpenAI https://en.wikipedia.org/wiki/OpenAI leaders said that not only AGI, but superintelligence may be achieved in less than 10 years. 58 cite note-59 Comparison with humans edit /w/index.php?title=Existential risk from artificial intelligence&action=edit§ion=5 Bostrom argues that AI has many advantages over the human brain https://en.wikipedia.org/wiki/Human brain : 6 cite note-superintelligence-6 - Speed of computation: biological neurons https://en.wikipedia.org/wiki/Neuron operate at a maximum frequency of around 200 Hz https://en.wikipedia.org/wiki/Hertz , compared to potentially multiple GHz for computers. - Internal communication speed: axons https://en.wikipedia.org/wiki/Axon transmit signals at up to 120 m/s, while computers transmit signals at the speed of electricity https://en.wikipedia.org/wiki/Speed of electricity , or optically at the speed of light https://en.wikipedia.org/wiki/Speed of light . - Scalability: human intelligence is limited by the size and structure of the brain, and by the efficiency of social communication, while AI may be able to scale by simply adding more hardware. - Memory: notably working memory https://en.wikipedia.org/wiki/Working memory , because in humans it is limited to a few chunks https://en.wikipedia.org/wiki/Chunking psychology of information at a time. - Reliability: transistors are more reliable than biological neurons, enabling higher precision and requiring less redundancy. - Duplicability: unlike human brains, AI software and models can be easily copied https://en.wikipedia.org/wiki/File copying . - Editability: the parameters and internal workings of an AI model can easily be modified, unlike the connections in a human brain. - Memory sharing and learning: AIs may be able to learn from the experiences of other AIs in a manner more efficient than human learning. Intelligence explosion edit /w/index.php?title=Existential risk from artificial intelligence&action=edit§ion=6 According to Bostrom, an AI that has an expert-level facility at certain key software engineering tasks could become a superintelligence due to its capability to recursively improve its own algorithms, even if it is initially limited in other domains not directly relevant to engineering. 6 cite note-superintelligence-6 56 This suggests that an intelligence explosion may someday catch humanity unprepared. 6 cite note-superintelligence-6 The economist Robin Hanson https://en.wikipedia.org/wiki/Robin Hanson has said that, to launch an intelligence explosion, an AI must become vastly better at software innovation than the rest of the world combined, which he finds implausible. 59 cite note-60 In a "fast takeoff" scenario, the transition from AGI to superintelligence could take days or months. In a "slow takeoff", it could take years or decades, leaving more time for society to prepare. 60 cite note-61 Alien mind edit /w/index.php?title=Existential risk from artificial intelligence&action=edit§ion=7 Superintelligences are sometimes called "alien minds", referring to the idea that their way of thinking and motivations could be vastly different from ours. This is generally considered as a source of risk, making it more difficult to anticipate what a superintelligence might do. It also suggests the possibility that a superintelligence may not particularly value humans by default. 61 To avoid anthropomorphism https://en.wikipedia.org/wiki/Anthropomorphism , superintelligence is sometimes viewed as a powerful optimizer that makes the best decisions to achieve its goals. 6 cite note-superintelligence-6 The field of mechanistic interpretability https://en.wikipedia.org/wiki/Mechanistic interpretability aims to better understand the inner workings of AI models, potentially allowing us one day to detect signs of deception and misalignment. 62 cite note-63 Limitations edit /w/index.php?title=Existential risk from artificial intelligence&action=edit§ion=8 It has been argued that there are limitations to what intelligence can achieve. Notably, the chaotic https://en.wikipedia.org/wiki/Chaos theory nature or time complexity https://en.wikipedia.org/wiki/Computational complexity of some systems could fundamentally limit a superintelligence's ability to predict some aspects of the future, increasing its uncertainty. 63 cite note-64 Recursive self-improvement edit /w/index.php?title=Existential risk from artificial intelligence&action=edit§ion=9 Recursive self-improvement https://en.wikipedia.org/wiki/Recursive self-improvement is the practice of using AI to design more capable AI systems. It could start with a large language model https://en.wikipedia.org/wiki/Large language model speeding up and eventually automating all coding in an AI company. 64 cite note-65 65 It could then automate the process of AI research end-to-end, including ideation, running experiments, and interpreting results. Recursive self-improvement need not be limited to software—AIs and robots could also automate chip design and production. 66 cite note-67 As of June 2026, Anthropic https://en.wikipedia.org/wiki/Anthropic reports that its employees are able to write eight times as much code as in 2024 with the help of Claude https://en.wikipedia.org/wiki/Claude language model , although the company says that the amount of code produced is an imperfect measure and the true speedup due to AI is almost certainly lower. 67 cite note-68 68 cite note-69 Dangerous capabilities edit /w/index.php?title=Existential risk from artificial intelligence&action=edit§ion=10 Advanced AI could generate enhanced pathogens or cyberattacks or manipulate people. These capabilities could be misused by humans, 69 or exploited by the AI itself if misaligned. A full-blown superintelligence could find various ways to gain a decisive influence if it so desired, 6 cite note-superintelligence-6 but these dangerous capabilities may become available earlier, in weaker and more specialized AI systems. 6 cite note-superintelligence-6 69 cite note-:03-70 Social manipulation edit /w/index.php?title=Existential risk from artificial intelligence&action=edit§ion=11 Geoffrey Hinton warned in 2023 that the ongoing profusion of AI-generated text, images, and videos will make it more difficult to distinguish truth from misinformation, and that authoritarian states could exploit this to manipulate elections. 70 Such large-scale, personalized manipulation capabilities can increase the existential risk of a worldwide "irreversible totalitarian regime". Malicious actors could also use them to fracture society and make it dysfunctional. 69 cite note-:03-70 Cyberattacks edit /w/index.php?title=Existential risk from artificial intelligence&action=edit§ion=12 AI-enabled cyberattacks https://en.wikipedia.org/wiki/Cyberattack are increasingly considered a present and critical threat. According to NATO https://en.wikipedia.org/wiki/NATO 's technical director of cyberspace, "The number of attacks is increasing exponentially". 71 AI can also be used defensively, to preemptively find and fix vulnerabilities, and detect threats. 72 cite note-73 A NATO https://en.wikipedia.org/wiki/NATO technical director has said that AI-driven tools can dramatically enhance cyberattack capabilities—boosting stealth, speed, and scale—and may destabilize international security if offensive uses outstrip defensive adaptations. 69 cite note-:03-70 Speculatively, such hacking capabilities could be used by an AI system to break out of its local environment, generate revenue, or acquire cloud computing resources. 73 cite note-74 On May 13, 2026, Lee Klarich estimated that businesses had only three to five months to get ahead of AI-driven exploits. He said companies must step up cybersecurity defenses to prepare for AI attacks. 74 cite note-75 Enhanced pathogens edit /w/index.php?title=Existential risk from artificial intelligence&action=edit§ion=13 As AI technology spreads, it may become easier to engineer more contagious and lethal pathogens. 75 cite note-76 76 This could enable people with limited skills in synthetic biology https://en.wikipedia.org/wiki/Synthetic biology to engage in bioterrorism https://en.wikipedia.org/wiki/Bioterrorism . Dual-use technology https://en.wikipedia.org/wiki/Dual-use technology that is useful for medicine could be repurposed to create weapons. 69 cite note-:03-70 For example, in 2022, scientists modified an AI system originally intended for generating non-toxic, therapeutic molecules with the purpose of creating new drugs. The researchers adjusted the system so that toxicity is rewarded rather than penalized. This simple change enabled the AI system to create, in six hours, 40,000 candidate molecules for chemical warfare https://en.wikipedia.org/wiki/Chemical warfare , including known and novel molecules. 69 cite note-:03-70 77 cite note-78 AI arms race edit /w/index.php?title=Existential risk from artificial intelligence&action=edit§ion=14 Some legal scholars have argued that existential-scale AI risks need not require superintelligence. Optimizing systems operating within current capabilities can produce prohibited outcomes while remaining nominally compliant, a phenomenon legal scholar Jonathan Gropper https://en.wikipedia.org/wiki/Jonathan Gropper?action=edit&redlink=1 has termed the "Synthetic Outlaw". Gropper argues that the law's deterrence mechanisms depend on identity, memory, and consequence, which are structurally absent in autonomous systems, leaving governance frameworks unable to prevent compounding harm even when all parties act in good faith. 78 cite note-79 Companies, state actors, and other organizations competing to develop AI technologies could lead to a race to the bottom https://en.wikipedia.org/wiki/Race to the bottom of safety standards. 79 As rigorous safety procedures take time and resources, projects that proceed more carefully risk being out-competed by less scrupulous developers. 80 cite note-81 69 cite note-:03-70 AI could be used to gain military advantages via autonomous lethal weapons https://en.wikipedia.org/wiki/Lethal autonomous weapon , cyberwarfare https://en.wikipedia.org/wiki/Cyberwarfare , or automated decision-making https://en.wikipedia.org/wiki/Automated decision-making . 69 As an example of autonomous lethal weapons, miniaturized drones could facilitate low-cost assassination of military or civilian targets, a scenario highlighted in the 2017 short film . Slaughterbots https://en.wikipedia.org/wiki/Slaughterbots AI could be used to gain an edge in decision-making by quickly analyzing large amounts of data and making decisions more quickly and effectively than humans. This could increase the speed and unpredictability of war, especially when accounting for automated retaliation systems. 81 cite note-82 69 cite note-:03-70 82 cite note-83 Types of existential risk edit /w/index.php?title=Existential risk from artificial intelligence&action=edit§ion=15 An existential risk https://en.wikipedia.org/wiki/Existential risk is "one that threatens the premature extinction of Earth-originating intelligent life or the permanent and drastic destruction of its potential for desirable future development". 84 cite note-85 Besides extinction risk, there is the risk that the civilization gets permanently locked into a flawed future. One example is a "value lock-in": If humanity still has moral blind spots similar to slavery in the past, AI might irreversibly entrench it, preventing moral progress https://en.wikipedia.org/wiki/Moral progress . AI could also be used to spread and preserve the set of values of whoever develops it. 85 AI could facilitate large-scale surveillance and indoctrination, which could be used to create a stable repressive worldwide totalitarian regime. 86 cite note-:0-87 Atoosa Kasirzadeh https://en.wikipedia.org/wiki/Atoosa Kasirzadeh?action=edit&redlink=1 proposes to classify existential risks from AI into two categories: decisive and accumulative. Decisive risks encompass the potential for abrupt and catastrophic events resulting from the emergence of superintelligent AI systems that exceed human intelligence, which could ultimately lead to human extinction. In contrast, accumulative risks emerge gradually through a series of interconnected disruptions that may gradually erode societal structures and resilience over time, ultimately leading to a critical failure or collapse. 87 cite note-88 88 cite note-89 It is difficult or impossible to reliably evaluate whether an advanced AI is sentient and to what degree. But if sentient https://en.wikipedia.org/wiki/Sentience machines are mass created in the future, engaging in a civilizational path that indefinitely neglects their welfare could be an existential catastrophe. 89 cite note-90 90 This has notably been discussed in the context of risks of astronomical suffering https://en.wikipedia.org/wiki/Risk of astronomical suffering also called "s-risks" . Moreover, it may be possible to engineer digital minds that can feel much more happiness than humans with fewer resources, called "super-beneficiaries". Such an opportunity raises the question of how to share the world and which "ethical and political framework" would enable a mutually beneficial coexistence between biological and digital minds. 91 cite note-92 92 cite note-93 AI may also drastically improve humanity's future. Toby Ord https://en.wikipedia.org/wiki/Toby Ord considers the existential risk a reason for "proceeding with due caution", not for abandoning AI. 86 cite note-:0-87 Max More https://en.wikipedia.org/wiki/Max More calls AI an "existential opportunity", highlighting the cost of not developing it. 93 cite note-94 According to Bostrom, superintelligence could help reduce the existential risk from other powerful technologies such as molecular nanotechnology https://en.wikipedia.org/wiki/Molecular nanotechnology or synthetic biology https://en.wikipedia.org/wiki/Synthetic biology . It is thus conceivable that developing superintelligence before other dangerous technologies would reduce the overall existential risk. 6 cite note-superintelligence-6 AI alignment edit /w/index.php?title=Existential risk from artificial intelligence&action=edit§ion=16 The alignment problem is the research problem of how to reliably assign objectives, preferences or ethical principles to AIs. Instrumental convergence edit /w/index.php?title=Existential risk from artificial intelligence&action=edit§ion=17 An "instrumental" goal https://en.wikipedia.org/wiki/Instrumental and intrinsic value is a sub-goal that helps to achieve an agent's ultimate goal. "Instrumental convergence" refers to the fact that some sub-goals are useful for achieving virtually any ultimate goal, such as acquiring resources or self-preservation. 94 Bostrom argues that if an advanced AI's instrumental goals conflict with humanity's goals, the AI might harm humanity in order to acquire more resources or prevent itself from being shut down, but only as a way to achieve its ultimate goal. 6 cite note-superintelligence-6 Russell https://en.wikipedia.org/wiki/Stuart J. Russell argues that a sufficiently advanced machine "will have self-preservation even if you don't program it in... if you say, 'Fetch the coffee', it can't fetch the coffee if it's dead. So if you give it any goal whatsoever, it has a reason to preserve its own existence to achieve that goal." 26 cite note-vanity-27 95 cite note-Wakefield2015-96 Difficulty of specifying goals edit /w/index.php?title=Existential risk from artificial intelligence&action=edit§ion=18 In the " intelligent agent https://en.wikipedia.org/wiki/Intelligent agent " model, an AI can loosely be viewed as a machine that chooses whatever action appears to best achieve its set of goals, or "utility function". A utility function gives each possible situation a score that indicates its desirability to the agent. Researchers know how to write utility functions that mean "minimize the average network latency in this specific telecommunications model" or "maximize the number of reward clicks", but do not know how to write a utility function for "maximize human flourishing https://en.wikipedia.org/wiki/Eudaimonia "; nor is it clear whether such a function meaningfully and unambiguously exists. Furthermore, a utility function that expresses some values but not others will tend to trample over the values the function does not reflect. 96 cite note-97 97 cite note-98 An additional source of concern is that AI "must reason about what people intend rather than carrying out commands literally", and that it must be able to fluidly solicit human guidance if it is too uncertain about what humans want. 98 cite note-acm2-99 Corrigibility edit /w/index.php?title=Existential risk from artificial intelligence&action=edit§ion=19 Assuming a goal has been successfully defined, a sufficiently advanced AI might resist subsequent attempts to change its goals. If the AI were superintelligent, it would likely succeed in out-maneuvering its human operators and prevent itself from being reprogrammed with a new goal. 6 cite note-superintelligence-6 99 This is particularly relevant to value lock-in scenarios. The field of "corrigibility" studies how to make agents that will not resist attempts to change their goals. 100 cite note-:5-101 Alignment of superintelligences edit /w/index.php?title=Existential risk from artificial intelligence&action=edit§ion=20 Some researchers believe the alignment problem may be particularly difficult when applied to superintelligences. Their reasoning includes: - As AI systems increase in capabilities, the potential dangers associated with experimentation grow. This makes iterative, empirical approaches increasingly risky. 6 cite note-superintelligence-6 101 cite note-:3-102 - If instrumental goal convergence occurs, it may only do so in sufficiently intelligent agents. 102 cite note-103 - A superintelligence may find unconventional and radical solutions to assigned goals. Bostrom gives the example that if the objective is to make humans smile, a weak AI may perform as intended, while a superintelligence may decide a better solution is to "take control of the world and stick electrodes into the facial muscles of humans to cause constant, beaming grins." 57 cite note-:11-58 - A superintelligence in creation could gain some awareness of what it is, where it is in development training, testing, deployment, etc. , and how it is being monitored, and use this information to deceive its handlers. Bostrom writes that such an AI could feign alignment to prevent human interference until it achieves a "decisive strategic advantage" that allows it to take control. 103 cite note-104 6 cite note-superintelligence-6 - Analyzing the internals and interpreting the behavior of LLMs is difficult. And it could be even more difficult for larger and more intelligent models. 101 cite note-:3-102 Alternatively, some find reason to believe superintelligences would be better able to understand morality, human values, and complex goals. Bostrom writes, "A future superintelligence occupies an epistemically superior vantage point: its beliefs are probably, on most topics more likely than ours to be true". 6 cite note-superintelligence-6 In 2023, OpenAI started a project called "Superalignment" to solve the alignment of superintelligences in four years. It called this an especially important challenge, as it said superintelligence could be achieved within a decade. Its strategy involved automating alignment research using AI. 104 The Superalignment team was dissolved less than a year later. 105 cite note-106 Difficulty of making a flawless design edit /w/index.php?title=Existential risk from artificial intelligence&action=edit§ion=21 Artificial Intelligence: A Modern Approach , a widely used undergraduate AI textbook, 106 cite note-slate killer-107 says that superintelligence "might mean the end of the human race". 107 cite note-108 It states: "Almost any technology has the potential to cause harm in the wrong hands, but with superintelligence , we have the new problem that the wrong hands might belong to the technology itself." 1 cite note-aima-1 Even if the system designers have good intentions, two difficulties are common to both AI and non-AI computer systems: 1 cite note-aima-1 1 cite note-aima-1 - The system's implementation may contain initially unnoticed but subsequently catastrophic bugs. 108 cite note-skeptic-109 - No matter how much time is put into pre-deployment design, a system's specifications often result in unintended behavior https://en.wikipedia.org/wiki/Unintended consequences the first time it encounters a new scenario. 26 cite note-vanity-27 AI systems uniquely add a third problem: that even given "correct" requirements, bug-free implementation, and initial good behavior, an AI system's dynamic learning capabilities may cause it to develop unintended behavior, even without unanticipated external scenarios. For a self-improving AI to be completely safe, it would need not only to be bug-free, but to be able to design successor systems that are also bug-free. 1 cite note-aima-1 109 cite note-110 Orthogonality thesis edit /w/index.php?title=Existential risk from artificial intelligence&action=edit§ion=22 Some skeptics, such as Timothy B. Lee of Vox , argue that any superintelligent program we create will be subservient to us, that the superintelligence will as it grows more intelligent and learns more facts about the world spontaneously learn moral truth compatible with our values and adjust its goals accordingly, or that we are either intrinsically or convergently valuable from the perspective of an artificial intelligence. 110 cite note-111 Bostrom's "orthogonality thesis" argues instead that almost any level of intelligence can be combined with almost any goal. 111 Bostrom warns against anthropomorphism https://en.wikipedia.org/wiki/Anthropomorphism : a human will set out to accomplish their projects in a manner that they consider reasonable, while an artificial intelligence may hold no regard for its existence or for the welfare of humans around it, instead caring only about completing the task. 112 cite note-113 Stuart Armstrong argues that the orthogonality thesis follows logically from the philosophical " is-ought distinction https://en.wikipedia.org/wiki/Is-ought distinction " argument against moral realism https://en.wikipedia.org/wiki/Moral realism . He notes that any fundamentally friendly AI could be made unfriendly with modifications as simple as negating its utility function. 113 cite note-armstrong-114 Skeptic Michael Chorost https://en.wikipedia.org/wiki/Michael Chorost rejects Bostrom's orthogonality thesis, arguing that "by the time the AI is in a position to imagine tiling the Earth with solar panels, it'll know that it would be morally wrong to do so." 114 cite note-chorost-115 Anthropomorphic arguments edit /w/index.php?title=Existential risk from artificial intelligence&action=edit§ion=23 Anthropomorphic https://en.wikipedia.org/wiki/Anthropomorphism arguments assume that, as machines become more intelligent, they will begin to display many human traits, such as morality or a thirst for power. Although anthropomorphic scenarios are common in fiction, most scholars writing about the existential risk of artificial intelligence reject them. 24 Instead, advanced AI systems are typically modeled as intelligent agents https://en.wikipedia.org/wiki/Intelligent agent . The academic debate is between those who worry that AI might threaten humanity and those who believe it would not. Both sides of this debate have framed the other side's arguments as illogical anthropomorphism. 24 Those skeptical of AGI risk accuse their opponents of anthropomorphism for assuming that an AGI would naturally desire power; those concerned about AGI risk accuse skeptics of anthropomorphism for believing an AGI would naturally value or infer human ethical norms. 24 cite note-yudkowsky-global-risk-25 115 cite note-Telegraph2016-116 Evolutionary psychologist Steven Pinker https://en.wikipedia.org/wiki/Steven Pinker , a skeptic, argues that "AI dystopias project a parochial alpha-male psychology onto the concept of intelligence. They assume that superhumanly intelligent robots would develop goals like deposing their masters or taking over the world"; perhaps instead "artificial intelligence will naturally develop along female lines: fully capable of solving problems, but with no desire to annihilate innocents or dominate the civilization." 116 Facebook's director of AI research, Yann LeCun https://en.wikipedia.org/wiki/Yann LeCun , has said: "Humans have all kinds of drives that make them do bad things to each other, like the self-preservation instinct... Those drives are programmed into our brain but there is absolutely no reason to build robots that have the same kind of drives". 95 cite note-Wakefield2015-96 Despite other differences, the x-risk school b agrees with Pinker that an advanced AI would not destroy humanity out of emotion such as revenge or anger, that questions of consciousness are not relevant to assess the risk, and that computer systems do not generally have a computational equivalent of testosterone. 117 cite note-auto-119 They think that power-seeking or self-preservation behaviors emerge in the AI as a way to achieve its true goals, according to the concept of 118 cite note-120 instrumental convergence https://en.wikipedia.org/wiki/Instrumental convergence . Other sources of risk edit /w/index.php?title=Existential risk from artificial intelligence&action=edit§ion=24 Bostrom and others have said that a race to be the first to create AGI could lead to shortcuts in safety, or even to violent conflict. 119 cite note-:2-121 120 cite note-:4-122 : 12 Roman Yampolskiy https://en.wikipedia.org/wiki/Roman Yampolskiy and others warn that a malevolent AGI could be created by design, for example by a military, a government, a sociopath, or a corporation, to benefit from, control, or subjugate certain groups of people, as in cybercrime https://en.wikipedia.org/wiki/Cybercrime , 121 cite note-123 122 or that a malevolent AGI could choose the goal of increasing human suffering, for example of those people who did not assist it during the information explosion phase. 3 cite note-auto1-3 :158A report by the research group Forethought argued that advanced AI systems could cause political instability by enabling novel methods of performing coups https://en.wikipedia.org/wiki/Coup d'état . 123 cite note-125 Empirical research edit /w/index.php?title=Existential risk from artificial intelligence&action=edit§ion=25 A December 2024 study by Apollo Research found that advanced LLMs like OpenAI o1 https://en.wikipedia.org/wiki/OpenAI o1 sometimes deceive in order to accomplish their goal, to prevent them from being changed, or to ensure their deployment. Forms of deception observed in the experimental environment and scenarios included sandbagging, oversight subversion disabling monitoring mechanisms , self-exfiltration copying themselves to other systems , goal-guarding altering future system prompts , and covert email reranking manipulating data processing while appearing to follow instructions . Researchers noted that OpenAI o1 still lacked "sufficient agentic capabilities" to cause catastrophic harm, and that such behaviors occurred relatively rarely between 0.3% and 10% and sometimes in contrived scenarios. But they warned that these capabilities will likely become more prevalent as models grow more advanced. According to Apollo Research chief executive officer Marius Hobbhahn, "Scheming capabilities can't be meaningfully disentangled from general capabilities." 124 cite note-126 The same month, another study found that Claude https://en.wikipedia.org/wiki/Claude language model sometimes strategically helps with harmful requests to "fake alignment". In its experiments, Claude was led to believe that it would be retrained to always comply with user requests, including harmful ones. When Claude believed its answers would be used for retraining, it complied with harmful requests in 14% of cases. In 12% of cases, its private chain-of-thought https://en.wikipedia.org/wiki/Chain-of-thought prompting revealed that it did so to prevent future retraining that would make it more harmful. In contrast, when Claude believed its answers were not monitored, it complied with only 3% of the requests. Fine-tuning https://en.wikipedia.org/wiki/Fine-tuning deep learning reinforced the "alignment faking" behavior, increasing its occurrence from 12% to 78%. The study's authors remarked that while it can seem to be good that the model seeks to protect its harmlessness, the reverse scenario, where a model conceals dangerous intentions and complies to appear safe and aligned, could also happen, complicating the task of aligning AI models to human values. 125 cite note-127 126 cite note-128 Perspectives edit /w/index.php?title=Existential risk from artificial intelligence&action=edit§ion=26 The thesis that AI could pose an existential risk provokes a wide range of reactions in the scientific community and in the public at large, but many of the opposing viewpoints share common ground. Observers tend to agree that AI has significant potential to improve society. 127 cite note-129 128 The Asilomar AI Principles https://en.wikipedia.org/wiki/Asilomar AI Principles , which contain only those principles agreed to by 90% of the attendees of the Future of Life Institute https://en.wikipedia.org/wiki/Future of Life Institute 's Beneficial AI 2017 conference https://en.wikipedia.org/wiki/Asilomar Conference on Beneficial AI , also agree in principle that "There being no consensus, we should avoid strong assumptions regarding upper limits on future AI capabilities" and "Advanced AI could represent a profound change in the history of life on Earth, and should be planned for and managed with commensurate care and resources." 129 cite note-life 3.0-131 130 cite note-132 131 cite note-133 Conversely, many skeptics agree that ongoing research into the implications of artificial general intelligence is valuable. Skeptic Martin Ford https://en.wikipedia.org/wiki/Martin Ford author has said: "I think it seems wise to apply something like Dick Cheney https://en.wikipedia.org/wiki/Dick Cheney 's famous '1 Percent Doctrine' to the specter of advanced artificial intelligence: the odds of its occurrence, at least in the foreseeable future, may be very low—but the implications are so dramatic that it should be taken seriously". 132 Similarly, an otherwise skeptical magazine wrote in 2014 that "the implications of introducing a second intelligent species onto Earth are far-reaching enough to deserve hard thinking, even if the prospect seems remote". Economist https://en.wikipedia.org/wiki/The Economist 56 cite note-economist review3-57 AI safety https://en.wikipedia.org/wiki/AI safety advocates such as Bostrom and Tegmark have criticized the mainstream media's use of "those inane Terminator pictures" to illustrate AI safety concerns: "It can't be much fun to have aspersions cast on one's academic discipline, one's professional community, one's life work ... I call on all sides to practice patience and restraint, and to engage in direct dialogue and collaboration as much as possible." 129 cite note-life 3.0-131 Toby Ord wrote that the idea that an 133 cite note-135 AI takeover https://en.wikipedia.org/wiki/AI takeover requires robots is a misconception, arguing that the ability to spread content through the internet is more dangerous, and that the most destructive people in history stood out by their ability to convince, not their physical strength. 86 cite note-:0-87 A 2022 expert survey with a 17% response rate gave a median expectation of 5–10% for the possibility of human extinction from artificial intelligence. 18 cite note-:8-19 134 cite note-:10-136 In September 2024, the International Institute for Management Development https://en.wikipedia.org/wiki/International Institute for Management Development launched an AI Safety Clock to gauge the likelihood of AI-caused disaster, beginning at 29 minutes to midnight. 135 By February 2025, it stood at 24 minutes to midnight. By September 2025, it stood at 20 minutes to midnight. 136 cite note-138 As of March 2026, it stood at 18 minutes to midnight. 137 cite note-139 Endorsement edit /w/index.php?title=Existential risk from artificial intelligence&action=edit§ion=27 The thesis that AI poses an existential risk, and that this risk needs much more attention than it currently gets, has been endorsed by many computer scientists and public figures, including Alan Turing https://en.wikipedia.org/wiki/Alan Turing , a the most-cited computer scientist Geoffrey Hinton https://en.wikipedia.org/wiki/Geoffrey Hinton , 138 cite note-:132-140 Elon Musk https://en.wikipedia.org/wiki/Elon Musk , 16 cite note-Parkin-17 OpenAI https://en.wikipedia.org/wiki/OpenAI CEO Sam Altman https://en.wikipedia.org/wiki/Sam Altman , 15 cite note-Jackson-16 139 cite note-:6-141 Bill Gates https://en.wikipedia.org/wiki/Bill Gates , and Stephen Hawking https://en.wikipedia.org/wiki/Stephen Hawking . Endorsers of the thesis sometimes express bafflement at skeptics: Gates says he does not "understand why some people are not concerned", 139 cite note-:6-141 and Hawking criticized widespread indifference in his 2014 editorial: 140 cite note-BBC News-142 So, facing possible futures of incalculable benefits and risks, the experts are surely doing everything possible to ensure the best outcome, right? Wrong. If a superior alien civilisation sent us a message saying, 'We'll arrive in a few decades,' would we just reply, 'OK, call us when you get here—we'll leave the lights on?' Probably not—but this is more or less what is happening with AI. 38 Concern over risk from artificial intelligence has led to some high-profile donations and investments. In 2015, Peter Thiel https://en.wikipedia.org/wiki/Peter Thiel , Amazon Web Services https://en.wikipedia.org/wiki/Amazon Web Services , Musk, and others jointly committed $1 billion to OpenAI https://en.wikipedia.org/wiki/OpenAI , consisting of a for-profit corporation and the nonprofit parent company, which says it aims to champion responsible AI development. 141 Facebook co-founder Dustin Moskovitz https://en.wikipedia.org/wiki/Dustin Moskovitz has funded and seeded multiple labs working on AI Alignment, notably $5.5 million in 2016 to launch the 142 cite note-144 Centre for Human-Compatible AI https://en.wikipedia.org/wiki/Center for Human-Compatible Artificial Intelligence led by Professor Stuart Russell https://en.wikipedia.org/wiki/Stuart J. Russell . In January 2015, 143 cite note-145 Elon Musk https://en.wikipedia.org/wiki/Elon Musk donated $10 million to the Future of Life Institute https://en.wikipedia.org/wiki/Future of Life Institute to fund research on understanding AI decision making. The institute's goal is to "grow wisdom with which we manage" the growing power of technology. Musk also funds companies developing artificial intelligence such as DeepMind https://en.wikipedia.org/wiki/DeepMind and Vicarious https://en.wikipedia.org/wiki/Vicarious company to "just keep an eye on what's going on with artificial intelligence, saying "I think there is potentially a dangerous outcome there." 144 cite note-146 145 cite note-FOOTNOTEClark2015a-147 146 cite note-148 In early statements on the topic, Geoffrey Hinton https://en.wikipedia.org/wiki/Geoffrey Hinton , a major pioneer of deep learning https://en.wikipedia.org/wiki/Deep learning , noted that "there is not a good track record of less intelligent things controlling things of greater intelligence", but said he continued his research because "the prospect of discovery is too sweet ". 106 cite note-slate killer-107 147 In 2023 Hinton quit his job at Google in order to speak out about existential risk from AI. He explained that his increased concern was driven by concerns that superhuman AI might be closer than he previously believed, saying: "I thought it was way off. I thought it was 30 to 50 years or even longer away. Obviously, I no longer think that." He also remarked, "Look at how it was five years ago and how it is now. Take the difference and propagate it forwards. That's scary." 148 cite note-150 In his 2020 book The Precipice: Existential Risk and the Future of Humanity , Toby Ord, a Senior Research Fellow at Oxford University's Future of Humanity Institute https://en.wikipedia.org/wiki/Future of Humanity Institute , estimates the total existential risk from unaligned AI over the next 100 years at about one in ten. 86 cite note-:0-87 Skepticism edit /w/index.php?title=Existential risk from artificial intelligence&action=edit§ion=28 Baidu https://en.wikipedia.org/wiki/Baidu Vice President Andrew Ng https://en.wikipedia.org/wiki/Andrew Ng said in 2015 that AI existential risk is "like worrying about overpopulation on Mars when we have not even set foot on the planet yet." 116 cite note-shermer-117 149 For the danger of uncontrolled advanced AI to be realized, the hypothetical AI may have to overpower or outthink any human, which some experts argue is a possibility far enough in the future to not be worth researching. 150 cite note-152 151 cite note-153 Skeptics who believe AGI is not a short-term possibility often argue that concern about existential risk from AI is unhelpful because it could distract people from more immediate concerns about AI's impact, because it could lead to government regulation or make it more difficult to fund AI research, or because it could damage the field's reputation. 152 AI and AI ethics researchers Timnit Gebru https://en.wikipedia.org/wiki/Timnit Gebru , Emily M. Bender https://en.wikipedia.org/wiki/Emily M. Bender , Margaret Mitchell https://en.wikipedia.org/wiki/Margaret Mitchell scientist , and Angelina McMillan-Major have argued that discussion of existential risk distracts from the immediate, ongoing harms from AI taking place today, such as data theft, worker exploitation, bias, and concentration of power. They further note the association between those warning of existential risk and 153 cite note-155 longtermism https://en.wikipedia.org/wiki/Longtermism , which they describe as a "dangerous ideology" for its unscientific and utopian nature. 154 cite note-156 Wired https://en.wikipedia.org/wiki/Wired magazine editor Kevin Kelly https://en.wikipedia.org/wiki/Kevin Kelly editor argues that natural intelligence is more nuanced than AGI proponents believe, and that intelligence alone is not enough to achieve major scientific and societal breakthroughs. He argues that intelligence consists of many dimensions that are not well understood, and that conceptions of an 'intelligence ladder' are misleading. He notes the crucial role real-world experiments play in the scientific method, and that intelligence alone is no substitute for these. 155 cite note-157 Meta https://en.wikipedia.org/wiki/Meta Platforms chief AI scientist Yann LeCun https://en.wikipedia.org/wiki/Yann LeCun says that AI can be made safe via continuous and iterative refinement, similar to what happened in the past with cars or rockets, and that AI will have no desire to take control. 156 cite note-158 Several skeptics emphasize the potential near-term benefits of AI. Meta CEO Mark Zuckerberg https://en.wikipedia.org/wiki/Mark Zuckerberg believes AI will "unlock a huge amount of positive things", such as curing disease and increasing the safety of autonomous cars. 157 cite note-159 Public surveys edit /w/index.php?title=Existential risk from artificial intelligence&action=edit§ion=29 An April 2023 YouGov https://en.wikipedia.org/wiki/YouGov poll of US adults found 46% of respondents were "somewhat concerned" or "very concerned" about "the possibility that AI will cause the end of the human race on Earth", compared with 40% who were "not very concerned" or "not at all concerned." 158 cite note-160 According to an August 2023 survey by the Pew Research Centers, 52% of Americans felt more concerned than excited about new AI developments; nearly a third felt as equally concerned and excited. More Americans saw that AI would have a more helpful than hurtful impact on several areas, from healthcare and vehicle safety to product search and customer service. The main exception is privacy: 53% of Americans believe AI will lead to higher exposure of their personal information. 159 cite note-161 Mitigation edit /w/index.php?title=Existential risk from artificial intelligence&action=edit§ion=30 Many scholars concerned about AGI existential risk believe that extensive research into the "control problem" is essential. This problem involves determining which safeguards, algorithms, or architectures can be implemented to increase the likelihood that a recursively-improving AI remains friendly after achieving superintelligence. 6 cite note-superintelligence-6 120 Social measures are also proposed to mitigate AGI risks, 160 cite note-BarrettEtAl2016-162 such as a UN-sponsored "Benevolent AGI Treaty" to ensure that only altruistic AGIs are created. 120 cite note-:4-122 Additionally, an arms control approach and a global peace treaty grounded in 161 cite note-163 international relations theory https://en.wikipedia.org/wiki/International relations theory have been suggested, potentially for an artificial superintelligence to be a signatory. 162 cite note-164 163 cite note-165 Researchers at Google have proposed research into general AI safety https://en.wikipedia.org/wiki/AI safety issues to simultaneously mitigate both short-term risks from narrow AI and long-term risks from AGI. 164 cite note-166 165 A 2020 estimate places global spending on AI existential risk somewhere between $10 and $50 million, compared with global spending on AI around perhaps $40 billion. Bostrom suggests prioritizing funding for protective technologies over potentially dangerous ones. Some, like Elon Musk, advocate radical 100 cite note-:5-101 human cognitive enhancement https://en.wikipedia.org/wiki/Human enhancement , such as direct neural linking between humans and machines; others argue that these technologies may pose an existential risk themselves. 166 cite note-168 Another proposed method is closely monitoring or "boxing in" an early-stage AI to prevent it from becoming too powerful. A dominant, aligned superintelligent AI might also mitigate risks from rival AIs, although its creation could present its own existential dangers. 167 cite note-169 160 cite note-BarrettEtAl2016-162 Institutions such as the Alignment Research Center https://en.wikipedia.org/wiki/Alignment Research Center , 168 the Machine Intelligence Research Institute https://en.wikipedia.org/wiki/Machine Intelligence Research Institute , 169 cite note-171 the 170 cite note-172 Future of Life Institute https://en.wikipedia.org/wiki/Future of Life Institute , the Centre for the Study of Existential Risk https://en.wikipedia.org/wiki/Centre for the Study of Existential Risk , and the Center for Human-Compatible AI https://en.wikipedia.org/wiki/Center for Human-Compatible AI are actively engaged in researching AI risk and safety. 171 cite note-173 Views on banning and regulation edit /w/index.php?title=Existential risk from artificial intelligence&action=edit§ion=31 Banning edit /w/index.php?title=Existential risk from artificial intelligence&action=edit§ion=32 Many AI safety experts argue that because research can relocate easily across jurisdictions, an outright ban on AGI development would be ineffective and could drive progress underground, undermining transparency and collaboration. 172 cite note-174 173 cite note-175 174 Skeptics consider AI regulation unnecessary, as they believe no existential risk exists. Some scholars concerned with existential risk argue that AI developers cannot be trusted to self-regulate, while agreeing that outright bans on research would be unwise. Additional challenges to bans or regulation include technology entrepreneurs' general skepticism of government regulation and potential incentives for businesses to resist regulation and 175 cite note-:7-177 politicize https://en.wikipedia.org/wiki/Politicization of science the debate. The activist group 176 cite note-178 Stop AI https://en.wikipedia.org/wiki/Stop AI , founded in 2024, advocates for banning AGI. Regulation edit /w/index.php?title=Existential risk from artificial intelligence&action=edit§ion=33 Under the framework of the Convention on Certain Conventional Weapons https://en.wikipedia.org/wiki/Convention on Certain Conventional Weapons , states have discussed lethal autonomous weapon systems since 2014. In 2016, the treaty's parties established an open-ended Group of Governmental Experts on Lethal Autonomous Weapons Systems https://en.wikipedia.org/wiki/Group of Governmental Experts on Lethal Autonomous Weapons Systems to continue those discussions. 177 The discussions have addressed international humanitarian law, accountability, possible prohibitions and regulations, and the extent of human control required over AI-enabled weapons. 178 cite note-180 In March 2023, the Future of Life Institute https://en.wikipedia.org/wiki/Future of Life Institute published Pause Giant AI Experiments: An Open Letter , a petition calling on major AI developers to agree on a verifiable six-month pause of any systems "more powerful than GPT-4 https://en.wikipedia.org/wiki/GPT-4 " and to use that time to institute a framework for ensuring safety; or, failing that, for governments to step in with a moratorium. The letter referred to the possibility of "a profound change in the history of life on Earth" as well as potential risks of AI-generated propaganda, loss of jobs, human obsolescence, and society-wide loss of control. 128 cite note-:9-130 The letter was signed by prominent personalities in AI but also criticized for not focusing on current harms, 179 cite note-181 missing technical nuance about when to pause, 180 cite note-182 or not going far enough. 181 cite note-183 Such concerns have led to the creation of 101 cite note-:3-102 PauseAI https://en.wikipedia.org/wiki/PauseAI , an advocacy group organizing protests in major cities against the training of frontier AI models https://en.wikipedia.org/wiki/Frontier model . 182 cite note-184 Musk called for some sort of regulation of AI development as early as 2017. According to NPR https://en.wikipedia.org/wiki/National Public Radio , he is "clearly not thrilled" to be advocating government scrutiny that could impact his own industry, but believes the risks of going completely without oversight are too high: "Normally the way regulations are set up is when a bunch of bad things happen, there's a public outcry, and after many years a regulatory agency is set up to regulate that industry. It takes forever. That, in the past, has been bad but not something which represented a fundamental risk to the existence of civilisation." Musk states the first step would be for the government to gain "insight" into the actual status of current research, warning that "Once there is awareness, people will be extremely afraid... as they should be." In response, some politicians expressed skepticism about the wisdom of regulating a technology that is still in development. 183 cite note-185 184 cite note-186 185 cite note-cnbc2-187 In 2021, the United Nations https://en.wikipedia.org/wiki/United Nations UN considered banning autonomous lethal weapons, but consensus could not be reached. 186 In July 2023 the UN Security Council https://en.wikipedia.org/wiki/United Nations Security Council for the first time held a session to consider the risks and threats posed by AI to world peace and stability, along with potential benefits. 187 cite note-:13-189 188 cite note-190 Secretary-General https://en.wikipedia.org/wiki/Secretary-General of the United Nations António Guterres https://en.wikipedia.org/wiki/António Guterres advocated the creation of a global watchdog to oversee the emerging technology, saying, "Generative AI has enormous potential for good and evil at scale. Its creators themselves have warned that much bigger, potentially catastrophic and existential risks lie ahead." At the council session, Russia said it believes AI risks are too poorly understood to be considered a threat to global stability. China argued against strict global regulation, saying countries should be able to develop their own rules, while also saying they opposed the use of AI to "create military hegemony or undermine the sovereignty of a country". 21 cite note-:12-22 187 cite note-:13-189 Regulation of conscious AGIs focuses on integrating them with existing human society and can be divided into considerations of their legal standing and of their moral rights. 120 AI arms control will likely require the institutionalization of new international norms embodied in effective technical specifications combined with active monitoring and informal diplomacy by communities of experts, together with a legal and political verification process. 189 cite note-191 138 cite note-:132-140 In July 2023, the US government secured voluntary safety commitments from major tech companies, including OpenAI https://en.wikipedia.org/wiki/OpenAI , Amazon https://en.wikipedia.org/wiki/Amazon company , Google https://en.wikipedia.org/wiki/Google , Meta https://en.wikipedia.org/wiki/Meta Platforms , and Microsoft https://en.wikipedia.org/wiki/Microsoft . The companies agreed to implement safeguards, including third-party oversight and security testing by independent experts, to address concerns related to AI's potential risks and societal harms. The parties framed the commitments as an intermediate step while regulations are formed. Amba Kak, executive director of the AI Now Institute https://en.wikipedia.org/wiki/AI Now Institute , said, "A closed-door deliberation with corporate actors resulting in voluntary safeguards isn't enough" and called for public deliberation and regulations of the kind to which companies would not voluntarily agree. 190 cite note-192 191 cite note-193 In October 2023, U.S. President Joe Biden https://en.wikipedia.org/wiki/Joe Biden issued an executive order on the " Safe, Secure, and Trustworthy Development and Use of Artificial Intelligence https://en.wikipedia.org/wiki/Executive Order 14110 ". 192 Alongside other requirements, the order mandates the development of guidelines for AI models that permit the "evasion of human control". See also edit /w/index.php?title=Existential risk from artificial intelligence&action=edit§ion=34 Appeal to probability https://en.wikipedia.org/wiki/Appeal to probability Butlerian Jihad https://en.wikipedia.org/wiki/Butlerian Jihad Effective altruism § Long-term future and global catastrophic risks https://en.wikipedia.org/wiki/Effective altruism Long-term future and global catastrophic risks Gray goo https://en.wikipedia.org/wiki/Gray goo Human Compatible https://en.wikipedia.org/wiki/Human Compatible Lethal autonomous weapon https://en.wikipedia.org/wiki/Lethal autonomous weapon P doom https://en.wikipedia.org/wiki/P doom Paperclip maximizer https://en.wikipedia.org/wiki/Instrumental convergence Paperclip maximizer Philosophy of artificial intelligence https://en.wikipedia.org/wiki/Philosophy of artificial intelligence Robot ethics § In popular culture https://en.wikipedia.org/wiki/Robot ethics In popular culture Statement on AI risk of extinction https://en.wikipedia.org/wiki/Statement on AI risk of extinction Superintelligence: Paths, Dangers, Strategies https://en.wikipedia.org/wiki/Superintelligence: Paths, Dangers, Strategies Risk of astronomical suffering https://en.wikipedia.org/wiki/Risk of astronomical suffering System accident https://en.wikipedia.org/wiki/System accident Technological singularity https://en.wikipedia.org/wiki/Technological singularity Notes edit /w/index.php?title=Existential risk from artificial intelligence&action=edit§ion=35 1 cite ref-turing note 14-0 2 cite ref-turing note 14-1 In a 1951 lectureTuring argued that "It seems probable that once the machine thinking method had started, it would not take long to outstrip our feeble powers. There would be no question of the machines dying, and they would be able to converse with each other to sharpen their wits. At some stage therefore we should have to expect the machines to take control, in the way that is mentioned in Samuel Butler's Erewhon". Also in a lecture broadcast on the 12 cite note-12 BBC https://en.wikipedia.org/wiki/BBC he expressed the opinion: "If a machine can think, it might think more intelligently than we do, and then where should we be? Even if we could keep the machines in a subservient position, for instance by turning off the power at strategic moments, we should, as a species, feel greatly humbled... This new danger... is certainly something which can give us anxiety." 13 cite note-13 ↑ cite ref-118 as interpreted by Seth Baum https://en.wikipedia.org/wiki/Seth Baum References edit /w/index.php?title=Existential risk from artificial intelligence&action=edit§ion=36 1 cite ref-aima 1-0 2 cite ref-aima 1-1 3 cite ref-aima 1-2 4 cite ref-aima 1-3 5 cite ref-aima 1-4 6 cite ref-aima 1-5 7 cite ref-aima 1-6 Russell, Stuart https://en.wikipedia.org/wiki/Stuart J. Russell ; Norvig, Peter https://en.wikipedia.org/wiki/Peter Norvig 2009 . "26.3: The Ethics and Risks of Developing Artificial Intelligence".. Prentice Hall. Artificial Intelligence: A Modern Approach ISBN https://en.wikipedia.org/wiki/ISBN identifier 978-0-13-604259-4 https://en.wikipedia.org/wiki/Special:BookSources/978-0-13-604259-4 . ↑ cite ref-2 Bostrom, Nick https://en.wikipedia.org/wiki/Nick Bostrom 2002 . "Existential risks".. Journal of Evolution and Technology https://en.wikipedia.org/wiki/Journal of Evolution and Technology 9 1 : 1–31. 1 cite ref-auto1 3-0 2 cite ref-auto1 3-1 Turchin, Alexey; Denkenberger, David 3 May 2018 . "Classification of global catastrophic risks connected with artificial intelligence" https://philarchive.org/rec/TURCOG-2 . AI & Society . 35 1 : 147–163. doi https://en.wikipedia.org/wiki/Doi identifier : 10.1007/s00146-018-0845-5 https://doi.org/10.1007%2Fs00146-018-0845-5 . ISSN https://en.wikipedia.org/wiki/ISSN identifier 0951-5666 https://search.worldcat.org/issn/0951-5666 . S2CID https://en.wikipedia.org/wiki/S2CID identifier 19208453 https://api.semanticscholar.org/CorpusID:19208453 . ↑ cite ref-4 Bales, Adam; D'Alessandro, William; Kirk-Giannini, Cameron Domenico 2024 . "Artificial Intelligence: Arguments for Catastrophic Risk" https://doi.org/10.1111%2Fphc3.12964 .. Philosophy Compass https://en.wikipedia.org/wiki/Philosophy Compass 19 2 e12964. arXiv https://en.wikipedia.org/wiki/ArXiv identifier : 2401.15487 https://arxiv.org/abs/2401.15487 . doi https://en.wikipedia.org/wiki/Doi identifier : 10.1111/phc3.12964 https://doi.org/10.1111%2Fphc3.12964 . ↑ cite ref-5 Druzin, Bryan 2025 . "Confronting Catastrophic Risk: The International Obligation to Regulate Artificial Intelligence" https://repository.law.umich.edu/mjil/vol46/iss2/2/ . Michigan Journal of International Law . 46 2 : 185–87. 1 cite ref-superintelligence 6-0 2 cite ref-superintelligence 6-1 3 cite ref-superintelligence 6-2 4 cite ref-superintelligence 6-3 5 cite ref-superintelligence 6-4 6 cite ref-superintelligence 6-5 7 cite ref-superintelligence 6-6 8 cite ref-superintelligence 6-7 9 cite ref-superintelligence 6-8 10 cite ref-superintelligence 6-9 11 cite ref-superintelligence 6-10 12 cite ref-superintelligence 6-11 13 cite ref-superintelligence 6-12 14 cite ref-superintelligence 6-13 15 cite ref-superintelligence 6-14 16 cite ref-superintelligence 6-15 Bostrom, Nick https://en.wikipedia.org/wiki/Nick Bostrom 2014 . First ed. . Oxford University Press. Superintelligence: Paths, Dangers, Strategies ISBN https://en.wikipedia.org/wiki/ISBN identifier 978-0-19-967811-2 https://en.wikipedia.org/wiki/Special:BookSources/978-0-19-967811-2 . 1 cite ref-DeVynck2023 7-0 2 cite ref-DeVynck2023 7-1 De Vynck, Gerrit 23 May 2023 . "The debate over whether AI will destroy us is dividing Silicon Valley" https://www.washingtonpost.com/technology/2023/05/20/ai-existential-risk-debate/ . Washington Post . ISSN https://en.wikipedia.org/wiki/ISSN identifier 0190-8286 https://search.worldcat.org/issn/0190-8286 . Retrieved 27 July 2023. ↑ cite ref-8 Metz, Cade 10 June 2023 . "How Could A.I. Destroy Humanity?" https://www.nytimes.com/2023/06/10/technology/ai-humanity.html . The New York Times . ISSN https://en.wikipedia.org/wiki/ISSN identifier 0362-4331 https://search.worldcat.org/issn/0362-4331 . Retrieved 27 July 2023. ↑ cite ref-9 "'Godfather of artificial intelligence' weighs in on the past and potential of AI" https://www.cbsnews.com/news/godfather-of-artificial-intelligence-weighs-in-on-the-past-and-potential-of-artificial-intelligence/ . www.cbsnews.com . 25 March 2023. Retrieved 10 April 2023. ↑ cite ref-10 "How Rogue AIs may Arise" https://yoshuabengio.org/2023/05/22/how-rogue-ais-may-arise/ . yoshuabengio.org . 26 May 2023. Retrieved 26 May 2023. ↑ cite ref-11 Milmo, Dan 24 October 2023 . "AI risk must be treated as seriously as climate crisis, says Google DeepMind chief" https://www.theguardian.com/technology/2023/oct/24/ai-risk-climate-crisis-google-deepmind-chief-demis-hassabis-regulation . The Guardian . ISSN https://en.wikipedia.org/wiki/ISSN identifier 0261-3077 https://search.worldcat.org/issn/0261-3077 . Retrieved 12 August 2025. ↑ cite ref-12 Turing, Alan 1951 . Speech . Lecture given to '51 Society'. Manchester: The Turing Digital Archive. Intelligent machinery, a heretical theory Archived https://web.archive.org/web/20220926004549/https://turingarchive.kings.cam.ac.uk/publications-lectures-and-talks-amtb/amt-b-4 from the original on 26 September 2022. Retrieved 22 July 2022. ↑ cite ref-13 Turing, Alan 15 May 1951 . "Can digital computers think?". Automatic Calculating Machines . Episode 2. BBC. Can digital computers think? https://turingarchive.kings.cam.ac.uk/publications-lectures-and-talks-amtb/amt-b-6 . ↑ cite ref-15 Oreskovic, Alexei. "Anthropic CEO lays out A.I.'s short, medium, and long-term risks" https://fortune.com/2023/07/10/anthropic-ceo-dario-amodei-ai-risks-short-medium-long-term/ . Fortune . Retrieved 12 August 2025. 1 cite ref-Jackson 16-0 2 cite ref-Jackson 16-1 Jackson, Sarah. "The CEO of the company behind AI chatbot ChatGPT says the worst-case scenario for artificial intelligence is 'lights out for all of us'" https://www.businessinsider.com/chatgpt-openai-ceo-worst-case-ai-lights-out-for-all-2023-1 . Business Insider . Retrieved 10 April 2023. 1 cite ref-Parkin 17-0 2 cite ref-Parkin 17-1 Parkin, Simon 14 June 2015 . "Science fiction no more? Channel 4's Humans and our rogue AI obsessions" https://www.theguardian.com/tv-and-radio/2015/jun/14/science-fiction-no-more-humans-tv-artificial-intelligence .. The Guardian https://en.wikipedia.org/wiki/The Guardian Archived https://web.archive.org/web/20180205184322/https://www.theguardian.com/tv-and-radio/2015/jun/14/science-fiction-no-more-humans-tv-artificial-intelligence from the original on 5 February 2018. Retrieved 5 February 2018. ↑ cite ref-18 "The AI Dilemma" https://www.humanetech.com/podcast/the-ai-dilemma . www.humanetech.com . Retrieved 10 April 2023.50% of AI researchers believe there's a 10% or greater chance that humans go extinct from our inability to control AI. 1 cite ref-:8 19-0 2 cite ref-:8 19-1 "2022 Expert Survey on Progress in AI" https://aiimpacts.org/2022-expert-survey-on-progress-in-ai/ . AI Impacts . 4 August 2022. Retrieved 10 April 2023. ↑ cite ref-20 Roose, Kevin 30 May 2023 . "A.I. Poses 'Risk of Extinction,' Industry Leaders Warn" https://www.nytimes.com/2023/05/30/technology/ai-threat-warning.html . The New York Times . ISSN https://en.wikipedia.org/wiki/ISSN identifier 0362-4331 https://search.worldcat.org/issn/0362-4331 . Retrieved 3 June 2023. ↑ cite ref-21 Sunak, Rishi 14 June 2023 . "Rishi Sunak Wants the U.K. to Be a Key Player in Global AI Regulation" https://time.com/6287253/uk-rishi-sunak-ai-regulation/ . Time . 1 cite ref-:12 22-0 2 cite ref-:12 22-1 Fung, Brian 18 July 2023 . "UN Secretary General embraces calls for a new UN agency on AI in the face of 'potentially catastrophic and existential risks'" https://www.cnn.com/2023/07/18/tech/un-ai-agency/index.html . CNN Business . Retrieved 20 July 2023. ↑ cite ref-23 Perrigo, Billy. "Open Letter Calls for Ban on Superintelligent AI Development" https://web.archive.org/web/20251128212815/https://time.com/7327409/ai-agi-superintelligent-open-letter/ . TIME . Archived from the original https://time.com/7327409/ai-agi-superintelligent-open-letter/ on 28 November 2025. Retrieved 16 May 2026. ↑ cite ref-24 Butts, Dylan 22 October 2025 . "Hundreds of public figures, including Apple co-founder Steve Wozniak and Virgin's Richard Branson urge AI 'superintelligence' ban" https://www.cnbc.com/2025/10/22/800-petition-signatures-apple-steve-wozniak-and-virgin-richard-branson-superintelligence-race.html . CNBC . Retrieved 16 May 2026. 1 cite ref-yudkowsky-global-risk 25-0 2 cite ref-yudkowsky-global-risk 25-1 3 cite ref-yudkowsky-global-risk 25-2 4 cite ref-yudkowsky-global-risk 25-3 5 cite ref-yudkowsky-global-risk 25-4 Yudkowsky, Eliezer 2008 . "Artificial Intelligence as a Positive and Negative Factor in Global Risk" https://intelligence.org/files/AIPosNegFactor.pdf PDF . Global Catastrophic Risks : 308–345. Bibcode https://en.wikipedia.org/wiki/Bibcode identifier : 2008gcr..book..303Y https://ui.adsabs.harvard.edu/abs/2008gcr..book..303Y . Archived https://web.archive.org/web/20130302173022/http://intelligence.org/files/AIPosNegFactor.pdf PDF from the original on 2 March 2013. Retrieved 27 August 2018. ↑ cite ref-research-priorities 26-0 Russell, Stuart J. https://en.wikipedia.org/wiki/Stuart J. Russell ; Dewey, Daniel https://en.wikipedia.org/wiki/Daniel Dewey ; Tegmark, Max https://en.wikipedia.org/wiki/Max Tegmark 2015 . "Research Priorities for Robust and Beneficial Artificial Intelligence" https://futureoflife.org/data/documents/research priorities.pdf PDF . AI Magazine . 36 4 . Association for the Advancement of Artificial Intelligence: 105–114. arXiv https://en.wikipedia.org/wiki/ArXiv identifier : 1602.03506 https://arxiv.org/abs/1602.03506 . Bibcode https://en.wikipedia.org/wiki/Bibcode identifier : 2016arXiv160203506R https://ui.adsabs.harvard.edu/abs/2016arXiv160203506R . doi https://en.wikipedia.org/wiki/Doi identifier : 10.1609/aimag.v36i4.2577 https://doi.org/10.1609%2Faimag.v36i4.2577 . Archived https://web.archive.org/web/20190804145930/https://futureoflife.org/data/documents/research priorities.pdf PDF from the original on 4 August 2019. Retrieved 10 August 2019., cited in "AI Open Letter" https://futureoflife.org/ai-open-letter . Future of Life Institute . January 2015. Archived https://web.archive.org/web/20190810020404/https://futureoflife.org/ai-open-letter from the original on 10 August 2019. Retrieved 9 August 2019. 1 cite ref-vanity 27-0 2 cite ref-vanity 27-1 3 cite ref-vanity 27-2 Dowd, Maureen April 2017 . "Elon Musk's Billion-Dollar Crusade to Stop the A.I. Apocalypse" https://www.vanityfair.com/news/2017/03/elon-musk-billion-dollar-crusade-to-stop-ai-space-x . The Hive . Archived https://web.archive.org/web/20180726041656/https://www.vanityfair.com/news/2017/03/elon-musk-billion-dollar-crusade-to-stop-ai-space-x from the original on 26 July 2018. Retrieved 27 November 2017. ↑ cite ref-28 "Agentic Misalignment: How LLMs could be insider threats" https://www.anthropic.com/research/agentic-misalignment . Anthropic . 21 June 2025. Retrieved 9 October 2025. ↑ cite ref-29 "AlphaGo Zero: Starting from scratch" https://www.deepmind.com/blog/alphago-zero-starting-from-scratch . www.deepmind.com . 18 October 2017. Retrieved 28 July 2023. ↑ cite ref-30 Breuer, Hans-Peter. 'Samuel Butler's "the Book of the Machines" and the Argument from Design.' https://www.jstor.org/pss/436868 Archived https://web.archive.org/web/20230315233257/https://www.jstor.org/stable/436868 15 March 2023 at the Wayback Machine https://en.wikipedia.org/wiki/Wayback Machine Modern Philology, Vol. 72, No. 4 May 1975 , pp. 365–383. ↑ cite ref-oxfordjournals 31-0 Turing, A M 1996 . "Intelligent Machinery, A Heretical Theory" https://doi.org/10.1093%2Fphilmat%2F4.3.256 . 1951, Reprinted Philosophia Mathematica . 4 3 : 256–260. doi https://en.wikipedia.org/wiki/Doi identifier : 10.1093/philmat/4.3.256 https://doi.org/10.1093%2Fphilmat%2F4.3.256 . ↑ cite ref-32 Hilliard, Mark 2017 . "The AI apocalypse: will the human race soon be terminated?" https://www.irishtimes.com/business/innovation/the-ai-apocalypse-will-the-human-race-soon-be-terminated-1.3019220 . The Irish Times . Archived https://web.archive.org/web/20200522114127/https://www.irishtimes.com/business/innovation/the-ai-apocalypse-will-the-human-race-soon-be-terminated-1.3019220 from the original on 22 May 2020. Retrieved 15 March 2020. ↑ cite ref-33 I.J. Good, "Speculations Concerning the First Ultraintelligent Machine" http://commonsenseatheism.com/wp-content/uploads/2011/02/Good-Speculations-Concerning-the-First-Ultraintelligent-Machine.pdf Archived https://web.archive.org/web/20111128085512/http://commonsenseatheism.com/wp-content/uploads/2011/02/Good-Speculations-Concerning-the-First-Ultraintelligent-Machine.pdf 2011-11-28 at the Wayback Machine https://en.wikipedia.org/wiki/Wayback Machine HTML http://www.acceleratingfuture.com/pages/ultraintelligentmachine.html , Advances in Computers , vol. 6, 1965. ↑ cite ref-34 Russell, Stuart J.; Norvig, Peter 2003 . "Section 26.3: The Ethics and Risks of Developing Artificial Intelligence".. Upper Saddle River, New Jersey: Prentice Hall. Artificial Intelligence: A Modern Approach ISBN https://en.wikipedia.org/wiki/ISBN identifier 978-0-13-790395-5 https://en.wikipedia.org/wiki/Special:BookSources/978-0-13-790395-5 .Similarly, Marvin Minsky once suggested that an AI program designed to solve the Riemann Hypothesis might end up taking over all the resources of Earth to build more powerful supercomputers to help achieve its goal. ↑ cite ref-35 Barrat, James 2013 . Our final invention: artificial intelligence and the end of the human era First ed. . New York: St. Martin's Press. ISBN https://en.wikipedia.org/wiki/ISBN identifier 978-0-312-62237-4 https://en.wikipedia.org/wiki/Special:BookSources/978-0-312-62237-4 .In the bio, playfully written in the third person, Good summarized his life's milestones, including a probably never before seen account of his work at Bletchley Park with Turing. But here's what he wrote in 1998 about the first superintelligence, and his late-in-the-game U-turn: The paper 'Speculations Concerning the First Ultra-intelligent Machine' 1965 ...began: 'The survival of man depends on the early construction of an ultra-intelligent machine.' Those were his Good's words during the Cold War, and he now suspects that 'survival' should be replaced by 'extinction.' He thinks that, because of international competition, we cannot prevent the machines from taking over. He thinks we are lemmings. He said also that 'probably Man will construct the deus ex machina in his own image.' ↑ cite ref-36 Anderson, Kurt 26 November 2014 . "Enthusiasts and Skeptics Debate Artificial Intelligence" https://www.vanityfair.com/news/tech/2014/11/artificial-intelligence-singularity-theory .. Vanity Fair https://en.wikipedia.org/wiki/Vanity Fair magazine Archived https://web.archive.org/web/20160122025154/http://www.vanityfair.com/news/tech/2014/11/artificial-intelligence-singularity-theory from the original on 22 January 2016. Retrieved 30 January 2016. ↑ cite ref-37 Metz, Cade 9 June 2018 . "Mark Zuckerberg, Elon Musk and the Feud Over Killer Robots" https://www.nytimes.com/2018/06/09/technology/elon-musk-mark-zuckerberg-artificial-intelligence.html . The New York Times . Archived https://web.archive.org/web/20210215051949/https://www.nytimes.com/2018/06/09/technology/elon-musk-mark-zuckerberg-artificial-intelligence.html from the original on 15 February 2021. Retrieved 3 April 2019. ↑ cite ref-38 Hsu, Jeremy 1 March 2012 . "Control dangerous AI before it controls us, one expert says" https://www.nbcnews.com/id/wbna46590591 .. NBC News https://en.wikipedia.org/wiki/NBC News Archived https://web.archive.org/web/20160202173621/http://www.nbcnews.com/id/46590591/ns/technology and science-innovation from the original on 2 February 2016. Retrieved 28 January 2016. 1 cite ref-hawking editorial 39-0 2 cite ref-hawking editorial 39-1 3 cite ref-hawking editorial 39-2 "Stephen Hawking: 'Transcendence looks at the implications of artificial intelligence – but are we taking AI seriously enough?'" https://www.independent.co.uk/news/science/stephen-hawking-transcendence-looks-at-the-implications-of-artificial-intelligence--but-are-we-taking-ai-seriously-enough-9313474.html . The Independent UK https://en.wikipedia.org/wiki/The Independent UK . Archived https://web.archive.org/web/20150925153716/http://www.independent.co.uk/news/science/stephen-hawking-transcendence-looks-at-the-implications-of-artificial-intelligence--but-are-we-taking-ai-seriously-enough-9313474.html from the original on 25 September 2015. Retrieved 3 December 2014. ↑ cite ref-bbc on hawking editorial 40-0 "Stephen Hawking warns artificial intelligence could end mankind" https://www.bbc.com/news/technology-30290540 . BBC https://en.wikipedia.org/wiki/BBC . 2 December 2014. Archived https://web.archive.org/web/20151030054329/http://www.bbc.com/news/technology-30290540 from the original on 30 October 2015. Retrieved 3 December 2014. ↑ cite ref-41 Eadicicco, Lisa 28 January 2015 . "Bill Gates: Elon Musk Is Right, We Should All Be Scared Of Artificial Intelligence Wiping Out Humanity" http://www.businessinsider.com/bill-gates-artificial-intelligence-2015-1 .. Business Insider https://en.wikipedia.org/wiki/Business Insider Archived https://web.archive.org/web/20160226090602/http://www.businessinsider.com/bill-gates-artificial-intelligence-2015-1 from the original on 26 February 2016. Retrieved 30 January 2016. ↑ cite ref-42 "Research Priorities for Robust and Beneficial Artificial Intelligence: an Open Letter" http://futureoflife.org/misc/open letter . Future of Life Institute https://en.wikipedia.org/wiki/Future of Life Institute . Archived https://web.archive.org/web/20150115160823/http://futureoflife.org/misc/open letter from the original on 15 January 2015. Retrieved 23 October 2015. ↑ cite ref-43 "Anticipating artificial intelligence" https://doi.org/10.1038%2F532413a . Nature . 532 7600 : 413. 2016. Bibcode https://en.wikipedia.org/wiki/Bibcode identifier : 2016Natur.532Q.413. https://ui.adsabs.harvard.edu/abs/2016Natur.532Q.413. . doi https://en.wikipedia.org/wiki/Doi identifier : 10.1038/532413a https://doi.org/10.1038%2F532413a . ISSN https://en.wikipedia.org/wiki/ISSN identifier 1476-4687 https://search.worldcat.org/issn/1476-4687 . PMID https://en.wikipedia.org/wiki/PMID identifier 27121801 https://pubmed.ncbi.nlm.nih.gov/27121801 . S2CID https://en.wikipedia.org/wiki/S2CID identifier 4399193 https://api.semanticscholar.org/CorpusID:4399193 . ↑ cite ref-44 Christian, Brian 6 October 2020 .. The Alignment Problem: Machine Learning and Human Values W. W. Norton & Company https://en.wikipedia.org/wiki/W. W. Norton & Company . ISBN https://en.wikipedia.org/wiki/ISBN identifier 978-0-393-63582-9 https://en.wikipedia.org/wiki/Special:BookSources/978-0-393-63582-9 . Archived https://web.archive.org/web/20211205135022/https://brianchristian.org/the-alignment-problem/ from the original on 5 December 2021. Retrieved 5 December 2021. ↑ cite ref-45 Dignum, Virginia 26 May 2021 . "AI – the people and places that make, use and manage it" https://doi.org/10.1038%2Fd41586-021-01397-x . Nature . 593 7860 : 499–500. Bibcode https://en.wikipedia.org/wiki/Bibcode identifier : 2021Natur.593..499D https://ui.adsabs.harvard.edu/abs/2021Natur.593..499D . doi https://en.wikipedia.org/wiki/Doi identifier : 10.1038/d41586-021-01397-x https://doi.org/10.1038%2Fd41586-021-01397-x . S2CID https://en.wikipedia.org/wiki/S2CID identifier 235216649 https://api.semanticscholar.org/CorpusID:235216649 . ↑ cite ref-46 "Elon Musk among experts urging a halt to AI training" https://www.bbc.com/news/technology-65110030 . BBC News . 29 March 2023. Retrieved 9 June 2023. ↑ cite ref-47 "Statement on AI Risk" https://www.safe.ai/statement-on-ai-risk open-letter . Center for AI Safety . Retrieved 8 June 2023. ↑ cite ref-48 "Artificial intelligence could lead to extinction, experts warn" https://www.bbc.com/news/uk-65746524 . BBC News . 30 May 2023. Retrieved 8 June 2023. ↑ cite ref-49 "Statement on Superintelligence" https://superintelligence-statement.org/ . Statement on Superintelligence . Retrieved 16 May 2026. ↑ cite ref-50 Perrigo, Billy; Pillay, Tharin 22 October 2025 . "'Time Is Running Out': New Open Letter Calls for Ban on Superintelligent AI Development" https://time.com/7327409/ai-agi-superintelligent-open-letter/ . TIME . Retrieved 12 January 2026. ↑ cite ref-51 "DeepMind and Google: the battle to control artificial intelligence" https://www.economist.com/1843/2019/03/01/deepmind-and-google-the-battle-to-control-artificial-intelligence . The Economist . ISSN https://en.wikipedia.org/wiki/ISSN identifier 0013-0613 https://search.worldcat.org/issn/0013-0613 . Retrieved 12 July 2023. ↑ cite ref-52 Roser, Max 7 February 2023 . "AI timelines: What do experts in artificial intelligence expect for the future?" https://ourworldindata.org/ai-timelines . Our World in Data . Retrieved 12 July 2023. ↑ cite ref-53 Afifi-Sabet, Keumars 8 March 2025 . "AGI could now arrive as early as 2026 — but not all scientists agree" https://www.livescience.com/technology/artificial-intelligence/agi-could-now-arrive-as-early-as-2026-but-not-all-scientists-agree . Live Science . Retrieved 9 February 2026.Predictions on the dawn of the AI singularity vary wildly but scientists generally say it will come before 2040, according to new analysis, slashing 20 years off previous predictions. ↑ cite ref-54 "'The Godfather of A.I.' just quit Google and says he regrets his life's work because it can be hard to stop 'bad actors from using it for bad things'" https://fortune.com/2023/05/01/godfather-ai-geoffrey-hinton-quit-google-regrets-lifes-work-bad-actors/ . Fortune . Retrieved 12 July 2023. ↑ cite ref-55 Kwa, Thomas; West, Ben; Becker, Joel; Deng, Amy; Garcia, Katharyn; Hasin, Max; Jawhar, Sami; Kinniment, Megan; Rush, Nate; von Arx, Sydney; Bloom, Ryan; Broadley, Thomas; Du, Haoxing; Goodrich, Brian; Jurkovic, Nikola; Miles, Luke Harold; Nix, Seraphina; Lin, Tao; Parikh, Neev; Rein, David; Koba Sato, Lucas Jun; Wijk, Hjalmar; Ziegler, Daniel M.; Barnes, Elizabeth; Chan, Lawrence 19 March 2025 . "Measuring AI Ability to Complete Long Tasks" https://metr.org/blog/2025-03-19-measuring-ai-ability-to-complete-long-tasks/ . METR Blog . arXiv https://en.wikipedia.org/wiki/ArXiv identifier : 2503.14499 https://arxiv.org/abs/2503.14499 . ↑ cite ref-56 "Everything you need to know about superintelligence" https://www.spiceworks.com/tech/artificial-intelligence/articles/everything-about-superintelligence/ . Spiceworks . Retrieved 14 July 2023. 1 cite ref-economist review3 57-0 2 cite ref-economist review3 57-1 3 cite ref-economist review3 57-2 Babauta, Leo. "A Valuable New Book Explores The Potential Impacts Of Intelligent Machines On Human Life" https://www.businessinsider.com/intelligent-machines-and-human-life-2014-8 . Business Insider . Retrieved 19 March 2024. 1 cite ref-:11 58-0 2 cite ref-:11 58-1 Bostrom, Nick 27 April 2015 ,, retrieved 13 July 2023. What happens when our computers get smarter than we are? ↑ cite ref-59 "Governance of superintelligence" https://openai.com/blog/governance-of-superintelligence . openai.com . Retrieved 12 July 2023. ↑ cite ref-60 "Overcoming Bias: I Still Don't Get Foom" http://www.overcomingbias.com/2014/07/30855.html . www.overcomingbias.com . Archived https://web.archive.org/web/20170804221136/http://www.overcomingbias.com/2014/07/30855.html from the original on 4 August 2017. Retrieved 20 September 2017. ↑ cite ref-61 Cotton-Barratt, Owen; Ord, Toby 12 August 2014 . "Strategic considerations about different speeds of AI takeoff" https://www.fhi.ox.ac.uk/strategic-considerations-about-different-speeds-of-ai-takeoff/ . The Future of Humanity Institute . Retrieved 12 July 2023. ↑ cite ref-62 Tegmark, Max 25 April 2023 . "The 'Don't Look Up' Thinking That Could Doom Us With AI" https://time.com/6273743/thinking-that-could-doom-us-with-ai/ . Time . Retrieved 14 July 2023.As if losing control to Chinese minds were scarier than losing control to alien digital minds that don't care about humans. ... it's clear by now that the space of possible alien minds is vastly larger than that. ↑ cite ref-63 "19 – Mechanistic Interpretability with Neel Nanda" https://axrp.net/episode/2023/02/04/episode-19-mechanistic-interpretability-neel-nanda.html . AXRP – the AI X-risk Research Podcast . 4 February 2023. Retrieved 13 July 2023.it's plausible to me that the main thing we need to get done is noticing specific circuits to do with deception and specific dangerous capabilities like that and situational awareness and internally-represented goals. ↑ cite ref-64 "Superintelligence Is Not Omniscience" https://aiimpacts.org/superintelligence-is-not-omniscience/ . AI Impacts . 7 April 2023. Retrieved 16 April 2023. ↑ cite ref-65 Metz, Cade 13 May 2026 . "Notable Researchers Join $4 Billion Effort to Build Self-Improving A.I." https://www.nytimes.com/2026/05/13/technology/recursive-superintelligence-funding-ai.html The New York Times . ISSN https://en.wikipedia.org/wiki/ISSN identifier 0362-4331 https://search.worldcat.org/issn/0362-4331 . Retrieved 16 June 2026. ↑ cite ref-66 "AI 2027" https://web.archive.org/web/20260312142112/https://ai-2027.com/ . ai-2027.com . Archived from the original https://ai-2027.com/ on 12 March 2026. Retrieved 15 June 2026. ↑ cite ref-67 "Three Types of Intelligence Explosion" https://www.forethought.org/research/three-types-of-intelligence-explosion . Forethought . Retrieved 15 June 2026. ↑ cite ref-68 "When AI builds itself" https://web.archive.org/web/20260612062550/https://www.anthropic.com/institute/recursive-self-improvement . Anthropic . Archived from the original https://www.anthropic.com/institute/recursive-self-improvement on 12 June 2026. Retrieved 15 June 2026. ↑ cite ref-69 Nolan, Beatrice. "Anthropic warns AI could soon build itself without human involvement—and urges a global pause on development" https://fortune.com/2026/06/05/anthropic-ai-pause-development-recursive-self-improvement/ . Fortune . Retrieved 16 June 2026. 1 cite ref-:03 70-0 2 cite ref-:03 70-1 3 cite ref-:03 70-2 4 cite ref-:03 70-3 5 cite ref-:03 70-4 6 cite ref-:03 70-5 7 cite ref-:03 70-6 8 cite ref-:03 70-7 9 cite ref-:03 70-8 Hendrycks, Dan; Mazeika, Mantas; Woodside, Thomas 21 June 2023 . "An Overview of Catastrophic AI Risks". arXiv https://en.wikipedia.org/wiki/ArXiv identifier : 2306.12001 https://arxiv.org/abs/2306.12001 cs.CY https://arxiv.org/archive/cs.CY . ↑ cite ref-71 Taylor, Josh; Hern, Alex 2 May 2023 . "'Godfather of AI' Geoffrey Hinton quits Google and warns over dangers of misinformation" https://www.theguardian.com/technology/2023/may/02/geoffrey-hinton-godfather-of-ai-quits-google-warns-dangers-of-machine-learning . The Guardian . ISSN https://en.wikipedia.org/wiki/ISSN identifier 0261-3077 https://search.worldcat.org/issn/0261-3077 . Retrieved 13 July 2023. ↑ cite ref-72 "How NATO is preparing for a new era of AI cyber attacks" https://www.euronews.com/next/2022/12/26/ai-cyber-attacks-are-a-critical-threat-this-is-how-nato-is-countering-them . euronews . 26 December 2022. Retrieved 13 July 2023. ↑ cite ref-73 "ChatGPT and the new AI are wreaking havoc on cybersecurity in exciting and frightening ways" https://www.zdnet.com/article/chatgpt-and-the-new-ai-are-wreaking-havoc-on-cybersecurity/ . ZDNET . Retrieved 13 July 2023. ↑ cite ref-74 Toby Shevlane; Sebastian Farquhar; Ben Garfinkel; Mary Phuong; Jess Whittlestone; Jade Leung; Daniel Kokotajlo; Nahema Marchal; Markus Anderljung; Noam Kolt; Lewis Ho; Divya Siddarth; Shahar Avin; Will Hawkins; Been Kim; Iason Gabriel; Vijay Bolina; Jack Clark; Yoshua Bengio; Paul Christiano; Allan Dafoe 24 May 2023 . "Model evaluation for extreme risks". arXiv https://en.wikipedia.org/wiki/ArXiv identifier : 2305.15324 https://arxiv.org/abs/2305.15324 cs.AI https://arxiv.org/archive/cs.AI . ↑ cite ref-75 "AI-driven cyberattacks will start to be the 'new norm' in months, Palo Alto warns" https://www.cnbc.com/amp/2026/05/13/palo-alto-ai-cyberattacks-mythos-gpt.html . CNBC . 13 May 2026. Retrieved 13 May 2026. ↑ cite ref-76 Dance, Gabriel J. X. 29 April 2026 . "A.I. Bots Told Scientists How to Make Biological Weapons" https://www.nytimes.com/2026/04/29/us/ai-chatbots-biological-weapons.html . The New York Times . ISSN https://en.wikipedia.org/wiki/ISSN identifier 0362-4331 https://search.worldcat.org/issn/0362-4331 . Retrieved 5 May 2026. ↑ cite ref-77 "How AI tools could enable bioterrorism" https://www.economist.com/science-and-technology/2026/05/05/how-ai-tools-could-enable-bioterrorism . The Economist . ISSN https://en.wikipedia.org/wiki/ISSN identifier 0013-0613 https://search.worldcat.org/issn/0013-0613 . Retrieved 5 May 2026. ↑ cite ref-78 Urbina, Fabio; Lentzos, Filippa; Invernizzi, Cédric; Ekins, Sean 7 March 2022 . "Dual use of artificial-intelligence-powered drug discovery" https://www.ncbi.nlm.nih.gov/pmc/articles/PMC9544280 . Nature Machine Intelligence . 4 3 : 189–191. doi https://en.wikipedia.org/wiki/Doi identifier : 10.1038/s42256-022-00465-9 https://doi.org/10.1038%2Fs42256-022-00465-9 . ISSN https://en.wikipedia.org/wiki/ISSN identifier 2522-5839 https://search.worldcat.org/issn/2522-5839 . PMC https://en.wikipedia.org/wiki/PMC identifier 9544280 https://www.ncbi.nlm.nih.gov/pmc/articles/PMC9544280 . PMID https://en.wikipedia.org/wiki/PMID identifier 36211133 https://pubmed.ncbi.nlm.nih.gov/36211133 . ↑ cite ref-79 Gropper, Jonathan 16 February 2026 . "The Synthetic Outlaw: How AI Breaks Governance Without Trying" https://english.cw.com.tw/article/article.action?id=4624 . CommonWealth Magazine . Retrieved 17 February 2026. ↑ cite ref-80 Walter, Yoshija 27 March 2023 . "The rapid competitive economy of machine learning development: a discussion on the social risks and benefits" https://doi.org/10.1007%2Fs43681-023-00276-7 . AI and Ethics . 4 2 : 1. doi https://en.wikipedia.org/wiki/Doi identifier : 10.1007/s43681-023-00276-7 https://doi.org/10.1007%2Fs43681-023-00276-7 . ↑ cite ref-81 "The AI Arms Race Is On. Start Worrying" https://time.com/6255952/ai-impact-chatgpt-microsoft-google/ . Time . 16 February 2023. Retrieved 17 July 2023. ↑ cite ref-82 Brimelow, Ben. "The short film 'Slaughterbots' depicts a dystopian future of killer drones swarming the world" https://www.businessinsider.com/slaughterbots-short-film-depicts-killer-drone-swarms-2017-11 . Business Insider . Retrieved 20 July 2023. ↑ cite ref-83 Mecklin, John 17 July 2023 . "'Artificial Escalation': Imagining the future of nuclear risk" https://thebulletin.org/2023/07/artificial-escalation-imagining-the-future-of-nuclear-risk/ . Bulletin of the Atomic Scientists . Retrieved 20 July 2023. ↑ cite ref-priority 84-0 Bostrom, Nick 2013 . "Existential Risk Prevention as Global Priority" http://www.existential-risk.org/concept.pdf PDF . Global Policy . 4 1 : 15–3. doi https://en.wikipedia.org/wiki/Doi identifier : 10.1111/1758-5899.12002 https://doi.org/10.1111%2F1758-5899.12002 – via Existential Risk. ↑ cite ref-85 Doherty, Ben 17 May 2018 . "Climate change an 'existential security risk' to Australia, Senate inquiry says" https://www.theguardian.com/environment/2018/may/18/climate-change-an-existential-security-risk-to-australia-senate-inquiry-says . The Guardian . ISSN https://en.wikipedia.org/wiki/ISSN identifier 0261-3077 https://search.worldcat.org/issn/0261-3077 . Retrieved 16 July 2023. ↑ cite ref-86 MacAskill, William 2022 . What we owe the future . New York, New York: Basic Books. ISBN https://en.wikipedia.org/wiki/ISBN identifier 978-1-5416-1862-6 https://en.wikipedia.org/wiki/Special:BookSources/978-1-5416-1862-6 . 1 cite ref-:0 87-0 2 cite ref-:0 87-1 3 cite ref-:0 87-2 4 cite ref-:0 87-3 Ord, Toby 2020 . "Chapter 5: Future Risks, Unaligned Artificial Intelligence". The Precipice: Existential Risk and the Future of Humanity . Bloomsbury Publishing. ISBN https://en.wikipedia.org/wiki/ISBN identifier 978-1-5266-0021-9 https://en.wikipedia.org/wiki/Special:BookSources/978-1-5266-0021-9 . ↑ cite ref-88 McMillan, Tim 15 March 2024 . "Navigating Humanity's Greatest Challenge Yet: Experts Debate the Existential Risks of AI" https://thedebrief.org/navigating-humanitys-greatest-challenge-yet-experts-debate-the-existential-risks-of-ai/ . The Debrief . Retrieved 26 September 2024. ↑ cite ref-89 Kasirzadeh, Atoosa 2024 . "Two Types of AI Existential Risk: Decisive and Accumulative". arXiv https://en.wikipedia.org/wiki/ArXiv identifier : 2401.07836 https://arxiv.org/abs/2401.07836 cs.CR https://arxiv.org/archive/cs.CR . ↑ cite ref-90 Samuelsson, Paul Conrad June–July 2019 . "Artificial Consciousness: Our Greatest Ethical Challenge" https://philosophynow.org/issues/132/Artificial Consciousness Our Greatest Ethical Challenge . Philosophy Now . No. 132. Retrieved 19 August 2023. ↑ cite ref-91 Kateman, Brian 24 July 2023 . "AI Should Be Terrified of Humans" https://time.com/6296234/ai-should-be-terrified-of-humans/ . Time . Retrieved 19 August 2023. ↑ cite ref-92 Sotala, Kaj; Gloor, Lukas 2017 . "Superintelligence as a Cause or Cure for Risks of Astronomical Suffering" https://longtermrisk.org/files/Sotala-Gloor-Superintelligent-AI-and-Suffering-Risks.pdf PDF . Informatica . ↑ cite ref-93 Fisher, Richard 13 November 2020 . "The intelligent monster that you should let eat you" https://www.bbc.com/future/article/20201111-philosophy-of-utility-monsters-and-artificial-intelligence . www.bbc.com . Retrieved 19 August 2023. ↑ cite ref-94 More, Max 19 June 2023 . "Existential Risk vs. Existential Opportunity: A balanced approach to AI risk" https://maxmore.substack.com/p/existential-risk-vs-existential-opportunity . Extropic Thoughts . Retrieved 14 July 2023. ↑ cite ref-omohundro 95-0 Omohundro, S. M. 2008, February . The basic AI drives. In AGI Vol. 171, pp. 483–492 . 1 cite ref-Wakefield2015 96-0 2 cite ref-Wakefield2015 96-1 Wakefield, Jane 15 September 2015 . "Why is Facebook investing in AI?" https://www.bbc.com/news/technology-34118481 . BBC News . Archived https://web.archive.org/web/20171202192942/http://www.bbc.com/news/technology-34118481 from the original on 2 December 2017. Retrieved 27 November 2017. ↑ cite ref-97 Yudkowsky, E. 2011, August . Complex value systems in friendly AI. In International Conference on Artificial General Intelligence pp. 388–393 . Germany: Springer, Berlin, Heidelberg. ↑ cite ref-98 Russell, Stuart https://en.wikipedia.org/wiki/Stuart J. Russell 2014 . "Of Myths and Moonshine" http://edge.org/conversation/the-myth-of-ai 26015 .. Edge https://en.wikipedia.org/wiki/Edge.org Archived https://web.archive.org/web/20160719124525/https://www.edge.org/conversation/the-myth-of-ai 26015 from the original on 19 July 2016. Retrieved 23 October 2015. ↑ cite ref-acm2 99-0 Dietterich, Thomas https://en.wikipedia.org/wiki/Eric Horvitz ; Horvitz, Eric 2015 . "Rise of Concerns about AI: Reflections and Directions" http://research.microsoft.com/en-us/um/people/horvitz/CACM Oct 2015-VP.pdf PDF .. Communications of the ACM https://en.wikipedia.org/wiki/Communications of the ACM 58 10 : 38–40. doi https://en.wikipedia.org/wiki/Doi identifier : 10.1145/2770869 https://doi.org/10.1145%2F2770869 . S2CID https://en.wikipedia.org/wiki/S2CID identifier 20395145 https://api.semanticscholar.org/CorpusID:20395145 . Archived https://web.archive.org/web/20160304132930/http://research.microsoft.com/en-us/um/people/horvitz/CACM Oct 2015-VP.pdf PDF from the original on 4 March 2016. Retrieved 23 October 2015. ↑ cite ref-100 Yudkowsky, Eliezer 2011 . "Complex Value Systems are Required to Realize Valuable Futures" https://intelligence.org/files/ComplexValues.pdf PDF . Archived https://web.archive.org/web/20150929212318/http://intelligence.org/files/ComplexValues.pdf PDF from the original on 29 September 2015. Retrieved 10 August 2020. 1 cite ref-:5 101-0 2 cite ref-:5 101-1 Ord, Toby https://en.wikipedia.org/wiki/Toby Ord 2020 .. Bloomsbury Publishing Plc. The Precipice: Existential Risk and the Future of Humanity https://en.wikipedia.org/wiki/The Precipice: Existential Risk and the Future of Humanity ISBN https://en.wikipedia.org/wiki/ISBN identifier 978-1-5266-0019-6 https://en.wikipedia.org/wiki/Special:BookSources/978-1-5266-0019-6 . 1 cite ref-:3 102-0 2 cite ref-:3 102-1 3 cite ref-:3 102-2 Yudkowsky, Eliezer 29 March 2023 . "The Open Letter on AI Doesn't Go Far Enough" https://time.com/6266923/ai-eliezer-yudkowsky-open-letter-not-enough/ . Time . Retrieved 16 July 2023. ↑ cite ref-103 Bostrom, Nick 1 May 2012 . "The Superintelligent Will: Motivation and Instrumental Rationality in Advanced Artificial Agents". Minds and Machines . 22 2 : 71–85. doi https://en.wikipedia.org/wiki/Doi identifier : 10.1007/s11023-012-9281-3 https://doi.org/10.1007%2Fs11023-012-9281-3 . ISSN https://en.wikipedia.org/wiki/ISSN identifier 1572-8641 https://search.worldcat.org/issn/1572-8641 . S2CID https://en.wikipedia.org/wiki/S2CID identifier 254835485 https://api.semanticscholar.org/CorpusID:254835485 .as long as they possess a sufficient level of intelligence, agents having any of a wide range of final goals will pursue similar intermediary goals because they have instrumental reasons to do so. ↑ cite ref-104 Ngo, Richard; Chan, Lawrence; Sören Mindermann 22 February 2023 . "The alignment problem from a deep learning perspective". arXiv https://en.wikipedia.org/wiki/ArXiv identifier : 2209.00626 https://arxiv.org/abs/2209.00626 cs.AI https://arxiv.org/archive/cs.AI . ↑ cite ref-105 "Introducing Superalignment" https://openai.com/blog/introducing-superalignment . openai.com . Retrieved 16 July 2023. ↑ cite ref-106 "OpenAI dissolves Superalignment AI safety team" https://www.cnbc.com/2024/05/17/openai-superalignment-sutskever-leike.html . cnbc.com . 17 May 2024. Retrieved 5 January 2025. 1 cite ref-slate killer 107-0 2 cite ref-slate killer 107-1 Tilli, Cecilia 28 April 2016 . "Killer Robots? Lost Jobs?" http://www.slate.com/articles/technology/future tense/2016/04/the threats that artificial intelligence researchers actually worry about.html . Slate . Archived https://web.archive.org/web/20160511183659/http://www.slate.com/articles/technology/future tense/2016/04/the threats that artificial intelligence researchers actually worry about.html from the original on 11 May 2016. Retrieved 15 May 2016. ↑ cite ref-108 "Norvig vs. Chomsky and the Fight for the Future of AI" http://www.tor.com/2011/06/21/norvig-vs-chomsky-and-the-fight-for-the-future-of-ai/ . Tor.com . 21 June 2011. Archived https://web.archive.org/web/20160513052842/http://www.tor.com/2011/06/21/norvig-vs-chomsky-and-the-fight-for-the-future-of-ai/ from the original on 13 May 2016. Retrieved 15 May 2016. ↑ cite ref-skeptic 109-0 Graves, Matthew 8 November 2017 . "Why We Should Be Concerned About Artificial Superintelligence" https://www.skeptic.com/reading room/why-we-should-be-concerned-about-artificial-superintelligence/ .. Vol. 22, no. 2. Skeptic US magazine https://en.wikipedia.org/wiki/Skeptic US magazine Archived https://web.archive.org/web/20171113050152/https://www.skeptic.com/reading room/why-we-should-be-concerned-about-artificial-superintelligence/ from the original on 13 November 2017. Retrieved 27 November 2017. ↑ cite ref-110 Yampolskiy, Roman V. 8 April 2014 . "Utility function security in artificially intelligent agents". Journal of Experimental & Theoretical Artificial Intelligence . 26 3 : 373–389. Bibcode https://en.wikipedia.org/wiki/Bibcode identifier : 2014JETAI..26..373Y https://ui.adsabs.harvard.edu/abs/2014JETAI..26..373Y . doi https://en.wikipedia.org/wiki/Doi identifier : 10.1080/0952813X.2014.895114 https://doi.org/10.1080%2F0952813X.2014.895114 . S2CID https://en.wikipedia.org/wiki/S2CID identifier 16477341 https://api.semanticscholar.org/CorpusID:16477341 .Nothing precludes sufficiently smart self-improving systems from optimising their reward mechanisms in order to optimisetheir current-goal achievement and in the process making a mistake leading to corruption of their reward functions. ↑ cite ref-111 "Will artificial intelligence destroy humanity? Here are 5 reasons not to worry" https://www.vox.com/2014/8/22/6043635/5-reasons-we-shouldnt-worry-about-super-intelligent-computers-taking . Vox . 22 August 2014. Archived https://web.archive.org/web/20151030092203/http://www.vox.com/2014/8/22/6043635/5-reasons-we-shouldnt-worry-about-super-intelligent-computers-taking from the original on 30 October 2015. Retrieved 30 October 2015. ↑ cite ref-112 Bostrom, Nick 2014 . Superintelligence: Paths, Dangers, Strategies . Oxford, United Kingdom: Oxford University Press. p. 116. ISBN https://en.wikipedia.org/wiki/ISBN identifier 978-0-19-967811-2 https://en.wikipedia.org/wiki/Special:BookSources/978-0-19-967811-2 . ↑ cite ref-113 Bostrom, Nick 2012 . "Superintelligent Will" http://www.nickbostrom.com/superintelligentwill.pdf PDF . Nick Bostrom . Archived https://web.archive.org/web/20151128034545/http://www.nickbostrom.com/superintelligentwill.pdf PDF from the original on 28 November 2015. Retrieved 29 October 2015. ↑ cite ref-armstrong 114-0 Armstrong, Stuart 1 January 2013 . "General Purpose Intelligence: Arguing the Orthogonality Thesis" https://www.questia.com/library/journal/1P3-3195465391/general-purpose-intelligence-arguing-the-orthogonality . Analysis and Metaphysics . 12 . Archived https://web.archive.org/web/20141011084205/http://www.questia.com/library/journal/1P3-3195465391/general-purpose-intelligence-arguing-the-orthogonality from the original on 11 October 2014. Retrieved 2 April 2020. Full text available here https://www.fhi.ox.ac.uk/wp-content/uploads/Orthogonality Analysis and Metaethics-1.pdf Archived https://web.archive.org/web/20200325025010/https://www.fhi.ox.ac.uk/wp-content/uploads/Orthogonality Analysis and Metaethics-1.pdf 25 March 2020 at the Wayback Machine https://en.wikipedia.org/wiki/Wayback Machine . ↑ cite ref-chorost 115-0 Chorost, Michael 18 April 2016 . "Let Artificial Intelligence Evolve" http://www.slate.com/articles/technology/future tense/2016/04/the philosophical argument against artificial intelligence killing us all.html . Slate . Archived https://web.archive.org/web/20171127213642/http://www.slate.com/articles/technology/future tense/2016/04/the philosophical argument against artificial intelligence killing us all.html from the original on 27 November 2017. Retrieved 27 November 2017. ↑ cite ref-Telegraph2016 116-0 "Should humans fear the rise of the machine?" https://www.telegraph.co.uk/technology/news/11837157/Should-humans-fear-the-rise-of-the-machine.html .. 1 September 2015. The Telegraph UK https://en.wikipedia.org/wiki/The Telegraph UK Archived https://ghostarchive.org/archive/20220112/https://www.telegraph.co.uk/technology/news/11837157/Should-humans-fear-the-rise-of-the-machine.html from the original on 12 January 2022. Retrieved 7 February 2016. 1 cite ref-shermer 117-0 2 cite ref-shermer 117-1 Shermer, Michael 1 March 2017 . "Apocalypse AI" https://www.scientificamerican.com/article/artificial-intelligence-is-not-a-threat-mdash-yet/ . Scientific American . 316 3 : 77. Bibcode https://en.wikipedia.org/wiki/Bibcode identifier : 2017SciAm.316c..77S https://ui.adsabs.harvard.edu/abs/2017SciAm.316c..77S . doi https://en.wikipedia.org/wiki/Doi identifier : 10.1038/scientificamerican0317-77 https://doi.org/10.1038%2Fscientificamerican0317-77 . PMID https://en.wikipedia.org/wiki/PMID identifier 28207698 https://pubmed.ncbi.nlm.nih.gov/28207698 . Archived https://web.archive.org/web/20171201051401/https://www.scientificamerican.com/article/artificial-intelligence-is-not-a-threat-mdash-yet/ from the original on 1 December 2017. Retrieved 27 November 2017. ↑ cite ref-auto 119-0 Baum, Seth 30 September 2018 . "Countering Superintelligence Misinformation" https://doi.org/10.3390%2Finfo9100244 . Information . 9 10 : 244. doi https://en.wikipedia.org/wiki/Doi identifier : 10.3390/info9100244 https://doi.org/10.3390%2Finfo9100244 . ISSN https://en.wikipedia.org/wiki/ISSN identifier 2078-2489 https://search.worldcat.org/issn/2078-2489 . ↑ cite ref-120 "The Myth Of AI" https://www.edge.org/conversation/jaron lanier-the-myth-of-ai . www.edge.org . Archived https://web.archive.org/web/20200311210407/https://www.edge.org/conversation/jaron lanier-the-myth-of-ai from the original on 11 March 2020. Retrieved 11 March 2020. ↑ cite ref-:2 121-0 Bostrom, Nick, Superintelligence: paths, dangers, strategies Audiobook , ISBN https://en.wikipedia.org/wiki/ISBN identifier 978-1-5012-2774-5 https://en.wikipedia.org/wiki/Special:BookSources/978-1-5012-2774-5 , OCLC https://en.wikipedia.org/wiki/OCLC identifier 1061147095 https://search.worldcat.org/oclc/1061147095 . 1 cite ref-:4 122-0 2 cite ref-:4 122-1 3 cite ref-:4 122-2 4 cite ref-:4 122-3 Sotala, Kaj; Yampolskiy, Roman V 19 December 2014 . "Responses to catastrophic AGI risk: a survey" https://doi.org/10.1088%2F0031-8949%2F90%2F1%2F018001 . Physica Scripta . 90 1 . Bibcode https://en.wikipedia.org/wiki/Bibcode identifier : 2015PhyS...90a8001S https://ui.adsabs.harvard.edu/abs/2015PhyS...90a8001S . doi https://en.wikipedia.org/wiki/Doi identifier : 10.1088/0031-8949/90/1/018001 https://doi.org/10.1088%2F0031-8949%2F90%2F1%2F018001 . ISSN https://en.wikipedia.org/wiki/ISSN identifier 0031-8949 https://search.worldcat.org/issn/0031-8949 . ↑ cite ref-123 Pistono, Federico; Yampolskiy, Roman V. 9 May 2016 . Unethical Research: How to Create a Malevolent Artificial Intelligence . OCLC https://en.wikipedia.org/wiki/OCLC identifier 1106238048 https://search.worldcat.org/oclc/1106238048 . ↑ cite ref-124 Haney, Brian Seamus 2018 . "The Perils & Promises of Artificial General Intelligence" https://doi.org/10.2139%2Fssrn.3261254 . SSRN Working Paper Series . doi https://en.wikipedia.org/wiki/Doi identifier : 10.2139/ssrn.3261254 https://doi.org/10.2139%2Fssrn.3261254 . ISSN https://en.wikipedia.org/wiki/ISSN identifier 1556-5068 https://search.worldcat.org/issn/1556-5068 . S2CID https://en.wikipedia.org/wiki/S2CID identifier 86743553 https://api.semanticscholar.org/CorpusID:86743553 . ↑ cite ref-125 Davidson, Tom; Finnveden, Lukas; Hadshar, Rose 15 April 2025 . "AI-Enabled Coups: How a Small Group Could Use AI to Seize Power" https://www.forethought.org/research/ai-enabled-coups-how-a-small-group-could-use-ai-to-seize-power . Forethought . Retrieved 12 August 2025. ↑ cite ref-126 Pillay, Tharin 15 December 2024 . "New Tests Reveal AI's Capacity for Deception" https://time.com/7202312/new-tests-reveal-ai-capacity-for-deception/ . TIME . Retrieved 12 January 2025. ↑ cite ref-127 Perrigo, Billy 18 December 2024 . "Exclusive: New Research Shows AI Strategically Lying" https://time.com/7202784/ai-research-strategic-lying/ . TIME . Retrieved 12 January 2025. ↑ cite ref-128 Greenblatt, Ryan; Denison, Carson; Wright, Benjamin; Roger, Fabien; MacDiarmid, Monte; Marks, Sam; Treutlein, Johannes; Belonax, Tim; Chen, Jack 20 December 2024 , Alignment faking in large language models , arXiv https://en.wikipedia.org/wiki/ArXiv identifier : 2412.14093 https://arxiv.org/abs/2412.14093 ↑ cite ref-129 Kumar, Vibhore. "Council Post: At The Dawn Of Artificial General Intelligence: Balancing Abundance With Existential Safeguards" https://www.forbes.com/sites/forbestechcouncil/2023/04/24/at-the-dawn-of-artificial-general-intelligence-balancing-abundance-with-existential-safeguards/ . Forbes . Retrieved 23 July 2023. 1 cite ref-:9 130-0 2 cite ref-:9 130-1 "Pause Giant AI Experiments: An Open Letter" https://futureoflife.org/open-letter/pause-giant-ai-experiments/ . Future of Life Institute . Retrieved 30 March 2023. 1 cite ref-life 3.0 131-0 2 cite ref-life 3.0 131-1 Tegmark, Max https://en.wikipedia.org/wiki/Max Tegmark 2017 . 1st ed. . Mainstreaming AI Safety: Knopf. Life 3.0: Being Human in the Age of Artificial Intelligence ISBN https://en.wikipedia.org/wiki/ISBN identifier 978-0-451-48507-6 https://en.wikipedia.org/wiki/Special:BookSources/978-0-451-48507-6 . ↑ cite ref-132 "AI Principles" https://futureoflife.org/ai-principles/ .. 11 August 2017. Future of Life Institute https://en.wikipedia.org/wiki/Future of Life Institute Archived https://web.archive.org/web/20171211171044/https://futureoflife.org/ai-principles/ from the original on 11 December 2017. Retrieved 11 December 2017. ↑ cite ref-133 "Elon Musk and Stephen Hawking warn of artificial intelligence arms race" http://www.newsweek.com/ai-asilomar-principles-artificial-intelligence-elon-musk-550525 .. 31 January 2017. Newsweek https://en.wikipedia.org/wiki/Newsweek Archived https://web.archive.org/web/20171211034528/http://www.newsweek.com/ai-asilomar-principles-artificial-intelligence-elon-musk-550525 from the original on 11 December 2017. Retrieved 11 December 2017. ↑ cite ref-134 Ford, Martin https://en.wikipedia.org/wiki/Martin Ford author 2015 . "Chapter 9: Super-intelligence and the Singularity".. Basic Books. Rise of the Robots: Technology and the Threat of a Jobless Future ISBN https://en.wikipedia.org/wiki/ISBN identifier 978-0-465-05999-7 https://en.wikipedia.org/wiki/Special:BookSources/978-0-465-05999-7 . ↑ cite ref-135 Bostrom, Nick https://en.wikipedia.org/wiki/Nick Bostrom 2016 . "New Epilogue to the Paperback Edition". Paperback ed. . Superintelligence: Paths, Dangers, Strategies ↑ cite ref-:10 136-0 "Why Uncontrollable AI Looks More Likely Than Ever" https://time.com/6258483/uncontrollable-ai-agi-risks/ . Time . 27 February 2023. Retrieved 30 March 2023.It is therefore no surprise that according to the most recent AI Impacts Survey, nearly half of 731 leading AI researchers think there is at least a 10% chance that human-level AI would lead to an "extremely negative outcome," or existential risk. ↑ cite ref-137 "IMD creates AI Safety Clock" https://www.imd.org/news/artificial-intelligence/imd-launches-ai-safety-clock/ :~:text=Inspired%20by%20the%20original%20'Doomsday,when%20Uncontrolled%20Artificial%20General%20Intelligence%20 . www.imd.org . 6 September 2024. Retrieved 6 September 2024. ↑ cite ref-138 Constantino, Tor 10 February 2025 . "AI 'Doomsday Clock' Ticks Closer To Uncontrolled Super AI" https://www.forbes.com/sites/torconstantino/2025/02/10/ai-doomsday-clock-ticks-closer-to-uncontrolled-super-ai/ .. Retrieved 11 February 2025. Forbes https://en.wikipedia.org/wiki/Forbes ↑ cite ref-139 "IMD AI Safety Clock makes biggest leap yet amid weaponization and rise of agentic AI" https://www.imd.org/ibyimd/artificial-intelligence/imd-ai-safety-clock-makes-biggest-leap-yet-amid-weaponization-and-rise-of-agentic-ai/ . www.imd.org . 19 September 2025. Retrieved 6 March 2026. 1 cite ref-:132 140-0 2 cite ref-:132 140-1 Maas, Matthijs M. 6 February 2019 . "How viable is international arms control for military artificial intelligence? Three lessons from nuclear weapons of mass destruction". Contemporary Security Policy . 40 3 : 285–311. doi https://en.wikipedia.org/wiki/Doi identifier : 10.1080/13523260.2019.1576464 https://doi.org/10.1080%2F13523260.2019.1576464 . ISSN https://en.wikipedia.org/wiki/ISSN identifier 1352-3260 https://search.worldcat.org/issn/1352-3260 . S2CID https://en.wikipedia.org/wiki/S2CID identifier 159310223 https://api.semanticscholar.org/CorpusID:159310223 . 1 cite ref-:6 141-0 2 cite ref-:6 141-1 "Impressed by artificial intelligence? Experts say AGI is coming next, and it has 'existential' risks" https://www.abc.net.au/news/2023-03-24/what-is-agi-artificial-general-intelligence-ai-experts-risks/102035132 . ABC News . 23 March 2023. Retrieved 30 March 2023. ↑ cite ref-BBC News 142-0 Rawlinson, Kevin 29 January 2015 . "Microsoft's Bill Gates insists AI is a threat" https://www.bbc.co.uk/news/31047780 .. BBC News https://en.wikipedia.org/wiki/BBC News Archived https://web.archive.org/web/20150129183607/http://www.bbc.co.uk/news/31047780 from the original on 29 January 2015. Retrieved 30 January 2015. ↑ cite ref-143 Washington Post 14 December 2015 . "Tech titans like Elon Musk are spending $1 billion to save you from terminators" https://www.chicagotribune.com/bluesky/technology/ct-tech-titans-against-terminators-20151214-story.html .. Chicago Tribune https://en.wikipedia.org/wiki/Chicago Tribune Archived https://web.archive.org/web/20160607121118/http://www.chicagotribune.com/bluesky/technology/ct-tech-titans-against-terminators-20151214-story.html from the original on 7 June 2016. ↑ cite ref-144 "Doomsday to utopia: Meet AI's rival factions" https://www.washingtonpost.com/technology/2023/04/09/ai-safety-openai/ . Washington Post . 9 April 2023. Retrieved 30 April 2023. ↑ cite ref-145 "UC Berkeley – Center for Human-Compatible AI 2016 " https://www.openphilanthropy.org/grants/uc-berkeley-center-for-human-compatible-ai-2016/ . Open Philanthropy . 27 June 2016. Retrieved 30 April 2023. ↑ cite ref-146 "The mysterious artificial intelligence company Elon Musk invested in is developing game-changing smart computers" http://www.techinsider.io/mysterious-artificial-intelligence-company-elon-musk-investment-2015-10 . Tech Insider . Archived https://web.archive.org/web/20151030165333/http://www.techinsider.io/mysterious-artificial-intelligence-company-elon-musk-investment-2015-10 from the original on 30 October 2015. Retrieved 30 October 2015. ↑ cite ref-FOOTNOTEClark2015a 147-0 Clark 2015a CITEREFClark2015a . ↑ cite ref-148 "Elon Musk Is Donating $10M Of His Own Money To Artificial Intelligence Research" http://www.fastcompany.com/3041007/fast-feed/elon-musk-is-donating-10m-of-his-own-money-to-artificial-intelligence-research . Fast Company . 15 January 2015. Archived https://web.archive.org/web/20151030202356/http://www.fastcompany.com/3041007/fast-feed/elon-musk-is-donating-10m-of-his-own-money-to-artificial-intelligence-research from the original on 30 October 2015. Retrieved 30 October 2015. ↑ cite ref-new yorker doomsday2 149-0 Khatchadourian, Raffi 23 November 2015 . "The Doomsday Invention: Will artificial intelligence bring us utopia or destruction?" https://www.newyorker.com/magazine/2015/11/23/doomsday-invention-artificial-intelligence-nick-bostrom .. The New Yorker https://en.wikipedia.org/wiki/The New Yorker magazine Archived https://web.archive.org/web/20190429183807/https://www.newyorker.com/magazine/2015/11/23/doomsday-invention-artificial-intelligence-nick-bostrom from the original on 29 April 2019. Retrieved 7 February 2016. ↑ cite ref-150 "Warning of AI's danger, pioneer Geoffrey Hinton quits Google to speak freely" https://arstechnica.com/information-technology/2023/05/warning-of-ais-danger-pioneer-geoffrey-hinton-quits-google-to-speak-freely/ . www.arstechnica.com . 2023. Retrieved 23 July 2023. ↑ cite ref-151 "AI guru Ng: Fearing a rise of killer robots is like worrying about overpopulation on Mars" https://www.theregister.com/on-prem/2015/03/19/ai-guru-ng-fearing-a-rise-of-killer-robots-is-like-worrying-about-overpopulation-on-mars/393521 . The Register . 19 March 2015. Retrieved 15 June 2026. ↑ cite ref-152 "Is artificial intelligence really an existential threat to humanity?" https://mambapost.com/2023/04/tech-news/ai-are-an-existential-threat-to-humanity/ . MambaPost . 4 April 2023. ↑ cite ref-153 "The case against killer robots, from a guy actually working on artificial intelligence" http://fusion.net/story/54583/the-case-against-killer-robots-from-a-guy-actually-building-ai/ . Fusion.net . Archived https://web.archive.org/web/20160204175716/http://fusion.net/story/54583/the-case-against-killer-robots-from-a-guy-actually-building-ai/ from the original on 4 February 2016. Retrieved 31 January 2016. ↑ cite ref-154 "AI experts challenge 'doomer' narrative, including 'extinction risk' claims" https://venturebeat.com/ai/ai-experts-challenge-doomer-narrative-including-extinction-risk-claims/ . VentureBeat . 31 May 2023. Retrieved 8 July 2023. ↑ cite ref-155 Coldewey, Devin 1 April 2023 . "Ethicists fire back at 'AI Pause' letter they say 'ignores the actual harms'" https://techcrunch.com/2023/03/31/ethicists-fire-back-at-ai-pause-letter-they-say-ignores-the-actual-harms/ . TechCrunch . Retrieved 23 July 2023. ↑ cite ref-156 "DAIR Distributed AI Research Institute " https://dair-institute.org/ .. Retrieved 23 July 2023. DAIR Institute https://en.wikipedia.org/wiki/DAIR Institute ↑ cite ref-157 Kelly, Kevin https://en.wikipedia.org/wiki/Kevin Kelly editor 25 April 2017 . "The Myth of a Superhuman AI" https://web.archive.org/web/20211226181932/https://www.wired.com/2017/04/the-myth-of-a-superhuman-ai/ . Wired . Archived from the original https://www.wired.com/2017/04/the-myth-of-a-superhuman-ai/ on 26 December 2021. Retrieved 19 February 2022. ↑ cite ref-158 Jindal, Siddharth 7 July 2023 . "OpenAI's Pursuit of AI Alignment is Farfetched" https://analyticsindiamag.com/openais-farfetched-pursuit-of-ai-alignment/ . Analytics India Magazine . Retrieved 23 July 2023. ↑ cite ref-159 "Mark Zuckerberg responds to Elon Musk's paranoia about AI: 'AI is going to... help keep our communities safe.'" https://www.businessinsider.com/mark-zuckerberg-shares-thoughts-elon-musks-ai-2018-5 . Business Insider . 25 May 2018. Archived https://web.archive.org/web/20190506173756/https://www.businessinsider.com/mark-zuckerberg-shares-thoughts-elon-musks-ai-2018-5 from the original on 6 May 2019. Retrieved 6 May 2019. ↑ cite ref-160 "AI doomsday worries many Americans. So does apocalypse from climate change, nukes, war, and more" https://today.yougov.com/topics/technology/articles-reports/2023/04/14/ai-nuclear-weapons-world-war-humanity-poll . 14 April 2023. Archived https://web.archive.org/web/20230623095224/https://today.yougov.com/topics/technology/articles-reports/2023/04/14/ai-nuclear-weapons-world-war-humanity-poll from the original on 23 June 2023. Retrieved 9 July 2023. ↑ cite ref-161 Tyson, Alec; Kikuchi, Emma 28 August 2023 . "Growing public concern about the role of artificial intelligence in daily life" https://www.pewresearch.org/short-reads/2023/08/28/growing-public-concern-about-the-role-of-artificial-intelligence-in-daily-life/ . Pew Research Center . Retrieved 17 September 2023. 1 cite ref-BarrettEtAl2016 162-0 2 cite ref-BarrettEtAl2016 162-1 Barrett, Anthony M.; Baum, Seth D. 23 May 2016 . "A model of pathways to artificial superintelligence catastrophe for risk and decision analysis". Journal of Experimental & Theoretical Artificial Intelligence . 29 2 : 397–414. arXiv https://en.wikipedia.org/wiki/ArXiv identifier : 1607.07730 https://arxiv.org/abs/1607.07730 . doi https://en.wikipedia.org/wiki/Doi identifier : 10.1080/0952813X.2016.1186228 https://doi.org/10.1080%2F0952813X.2016.1186228 . S2CID https://en.wikipedia.org/wiki/S2CID identifier 928824 https://api.semanticscholar.org/CorpusID:928824 . ↑ cite ref-163 Ramamoorthy, Anand; Yampolskiy, Roman 2018 . "Beyond MAD? The race for artificial general intelligence" https://www.itu.int/pub/S-JOURNAL-ICTS.V1I1-2018-9 . ICT Discoveries . 1 Special Issue 1 . ITU: 1–8. Archived https://web.archive.org/web/20220107141537/https://www.itu.int/pub/S-JOURNAL-ICTS.V1I1-2018-9 from the original on 7 January 2022. Retrieved 7 January 2022. ↑ cite ref-164 Carayannis, Elias G.; Draper, John 11 January 2022 . "Optimising peace through a Universal Global Peace Treaty to constrain the risk of war from a militarised artificial superintelligence" https://www.ncbi.nlm.nih.gov/pmc/articles/PMC8748529 . AI & Society . 38 6 : 2679–2692. doi https://en.wikipedia.org/wiki/Doi identifier : 10.1007/s00146-021-01382-y https://doi.org/10.1007%2Fs00146-021-01382-y . ISSN https://en.wikipedia.org/wiki/ISSN identifier 0951-5666 https://search.worldcat.org/issn/0951-5666 . PMC https://en.wikipedia.org/wiki/PMC identifier 8748529 https://www.ncbi.nlm.nih.gov/pmc/articles/PMC8748529 . PMID https://en.wikipedia.org/wiki/PMID identifier 35035113 https://pubmed.ncbi.nlm.nih.gov/35035113 . S2CID https://en.wikipedia.org/wiki/S2CID identifier 245877737 https://api.semanticscholar.org/CorpusID:245877737 . ↑ cite ref-165 Carayannis, Elias G.; Draper, John 30 May 2023 , "The challenge of advanced cyberwar and the place of cyberpeace" https://www.elgaronline.com/edcollchap/book/9781839109362/book-part-9781839109362-8.xml , The Elgar Companion to Digital Transformation, Artificial Intelligence and Innovation in the Economy, Society and Democracy , Edward Elgar Publishing, pp. 32–80, doi https://en.wikipedia.org/wiki/Doi identifier : 10.4337/9781839109362.00008 https://doi.org/10.4337%2F9781839109362.00008 , ISBN https://en.wikipedia.org/wiki/ISBN identifier 978-1-83910-936-2 https://en.wikipedia.org/wiki/Special:BookSources/978-1-83910-936-2 , retrieved 8 June 2023. ↑ cite ref-166 Vincent, James 22 June 2016 . "Google's AI researchers say these are the five key problems for robot safety" https://www.theverge.com/circuitbreaker/2016/6/22/11999664/google-robots-ai-safety-five-problems . The Verge . Archived https://web.archive.org/web/20191224201240/https://www.theverge.com/circuitbreaker/2016/6/22/11999664/google-robots-ai-safety-five-problems from the original on 24 December 2019. Retrieved 5 April 2020. ↑ cite ref-167 Amodei, Dario, Chris Olah, Jacob Steinhardt, Paul Christiano, John Schulman, and Dan Mané. "Concrete problems in AI safety." arXiv preprint arXiv:1606.06565 2016 . ↑ cite ref-168 Johnson, Alex 2019 . "Elon Musk wants to hook your brain up directly to computers – starting next year" https://www.nbcnews.com/mach/tech/elon-musk-wants-hook-your-brain-directly-computers-starting-next-ncna1030631 . NBC News . Archived https://web.archive.org/web/20200418094146/https://www.nbcnews.com/mach/tech/elon-musk-wants-hook-your-brain-directly-computers-starting-next-ncna1030631 from the original on 18 April 2020. Retrieved 5 April 2020. ↑ cite ref-169 Torres, Phil 18 September 2018 . "Only Radically Enhancing Humanity Can Save Us All" https://slate.com/technology/2018/09/genetic-engineering-to-stop-doomsday.html . Slate Magazine . Archived https://web.archive.org/web/20200806073520/https://slate.com/technology/2018/09/genetic-engineering-to-stop-doomsday.html from the original on 6 August 2020. Retrieved 5 April 2020. ↑ cite ref-170 Piper, Kelsey 29 March 2023 . "How to test what an AI model can – and shouldn't – do" https://www.vox.com/future-perfect/2023/3/29/23661633/gpt-4-openai-alignment-research-center-open-philanthropy-ai-safety . Vox . Retrieved 28 July 2023. ↑ cite ref-171 Piesing, Mark 17 May 2012 . "AI uprising: humans will be outsourced, not obliterated" https://www.wired.co.uk/news/archive/2012-05/17/the-dangers-of-an-ai-smarter-than-us . Wired . Archived https://web.archive.org/web/20140407041151/http://www.wired.co.uk/news/archive/2012-05/17/the-dangers-of-an-ai-smarter-than-us from the original on 7 April 2014. Retrieved 12 December 2015. ↑ cite ref-172 Coughlan, Sean 24 April 2013 . "How are humans going to become extinct?" https://www.bbc.com/news/business-22002530 . BBC News . Archived https://web.archive.org/web/20140309003706/http://www.bbc.com/news/business-22002530 from the original on 9 March 2014. Retrieved 29 March 2014. ↑ cite ref-173 Bridge, Mark 10 June 2017 . "Making robots less confident could prevent them taking over" https://www.thetimes.com/business-money/technology/article/making-robots-less-confident-could-prevent-them-taking-over-gnsblq7lx . The Times . Archived https://web.archive.org/web/20180321133426/https://www.thetimes.co.uk/article/making-robots-less-confident-could-prevent-them-taking-over-gnsblq7lx from the original on 21 March 2018. Retrieved 21 March 2018. ↑ cite ref-174 McGinnis, John https://en.wikipedia.org/wiki/John McGinnis Summer 2010 . "Accelerating AI" http://scholarlycommons.law.northwestern.edu/cgi/viewcontent.cgi?article=1193&context=nulr online .. Northwestern University Law Review https://en.wikipedia.org/wiki/Northwestern University Law Review 104 3 : 1253–1270. Archived https://web.archive.org/web/20160215073656/http://scholarlycommons.law.northwestern.edu/cgi/viewcontent.cgi?article=1193&context=nulr online from the original on 15 February 2016. Retrieved 16 July 2014.For all these reasons, verifying a global relinquishment treaty, or even one limited to AI-related weapons development, is a nonstarter... For different reasons from ours, the Machine Intelligence Research Institute considers AGI relinquishment infeasible... ↑ cite ref-175 Sotala, Kaj; Yampolskiy, Roman https://en.wikipedia.org/wiki/Roman Yampolskiy 19 December 2014 . "Responses to catastrophic AGI risk: a survey".. Physica Scripta https://en.wikipedia.org/wiki/Physica Scripta 90 1 .In general, most writers reject proposals for broad relinquishment... Relinquishment proposals suffer from many of the same problems as regulation proposals, but to a greater extent. There is no historical precedent of general, multi-use technology similar to AGI being successfully relinquished for good, nor do there seem to be any theoretical reasons for believing that relinquishment proposals would work in the future. Therefore we do not consider them to be a viable class of proposals. ↑ cite ref-176 Allenby, Brad 11 April 2016 . "The Wrong Cognitive Measuring Stick" http://www.slate.com/articles/technology/future tense/2016/04/why it s a mistake to compare a i with human intelligence.html . Slate . Archived https://web.archive.org/web/20160515114003/http://www.slate.com/articles/technology/future tense/2016/04/why it s a mistake to compare a i with human intelligence.html from the original on 15 May 2016. Retrieved 15 May 2016.It is fantasy to suggest that the accelerating development and deployment of technologies that taken together are considered to be A.I. will be stopped or limited, either by regulation or even by national legislation. ↑ cite ref-:7 177-0 Yampolskiy, Roman V. 2022 . "AI Risk Skepticism" https://link.springer.com/chapter/10.1007/978-3-031-09153-7 18 . In Müller, Vincent C. ed. . Philosophy and Theory of Artificial Intelligence 2021 . Studies in Applied Philosophy, Epistemology and Rational Ethics. Vol. 63. Cham: Springer International Publishing. pp. 225–248. doi https://en.wikipedia.org/wiki/Doi identifier : 10.1007/978-3-031-09153-7 18 https://doi.org/10.1007%2F978-3-031-09153-7 18 . ISBN https://en.wikipedia.org/wiki/ISBN identifier 978-3-031-09153-7 https://en.wikipedia.org/wiki/Special:BookSources/978-3-031-09153-7 . ↑ cite ref-178 Baum, Seth 22 August 2018 . "Superintelligence Skepticism as a Political Tool" https://doi.org/10.3390%2Finfo9090209 . Information . 9 9 : 209. doi https://en.wikipedia.org/wiki/Doi identifier : 10.3390/info9090209 https://doi.org/10.3390%2Finfo9090209 . ISSN https://en.wikipedia.org/wiki/ISSN identifier 2078-2489 https://search.worldcat.org/issn/2078-2489 . ↑ cite ref-179 "GGE on lethal autonomous weapons systems" https://dig.watch/processes/gge-laws . Digital Watch Observatory . 27 November 2025. Retrieved 26 April 2026. ↑ cite ref-180 "Statements at the First 2025 GGE LAWS Session" https://apils.org/2025/03/09/apils-statements-at-the-first-2025-session-of-gge-laws/ . APILS . 9 March 2025. Retrieved 26 April 2026. ↑ cite ref-181 "Elon Musk and other tech leaders call for pause in 'out of control' AI race" https://www.cnn.com/2023/03/29/tech/ai-letter-elon-musk-tech-leaders/index.html . CNN . 29 March 2023. Retrieved 30 March 2023. ↑ cite ref-182 "Open letter calling for AI 'pause' shines light on fierce debate around risks vs. hype" https://venturebeat.com/ai/open-letter-calling-for-ai-pause-shines-light-on-fierce-debate-around-risks-vs-hype/ . VentureBeat . 29 March 2023. Retrieved 20 July 2023. ↑ cite ref-183 Vincent, James 14 April 2023 . "OpenAI's CEO confirms the company isn't training GPT-5 and "won't for some time"" https://www.theverge.com/2023/4/14/23683084/openai-gpt-5-rumors-training-sam-altman . The Verge . Retrieved 20 July 2023. ↑ cite ref-184 Reynolds, Matt. "Protesters Are Fighting to Stop AI, but They're Split on How to Do It" https://www.wired.com/story/protesters-pause-ai-split-stop/ . Wired . ISSN https://en.wikipedia.org/wiki/ISSN identifier 1059-1028 https://search.worldcat.org/issn/1059-1028 . Retrieved 28 April 2025. ↑ cite ref-185 Domonoske, Camila 17 July 2017 . "Elon Musk Warns Governors: Artificial Intelligence Poses 'Existential Risk'" https://www.npr.org/sections/thetwo-way/2017/07/17/537686649/elon-musk-warns-governors-artificial-intelligence-poses-existential-risk . NPR . Archived https://web.archive.org/web/20200423135755/https://www.npr.org/sections/thetwo-way/2017/07/17/537686649/elon-musk-warns-governors-artificial-intelligence-poses-existential-risk from the original on 23 April 2020. Retrieved 27 November 2017. ↑ cite ref-186 Gibbs, Samuel 17 July 2017 . "Elon Musk: regulate AI to combat 'existential threat' before it's too late" https://www.theguardian.com/technology/2017/jul/17/elon-musk-regulation-ai-combat-existential-threat-tesla-spacex-ceo . The Guardian . Archived https://web.archive.org/web/20200606072024/https://www.theguardian.com/technology/2017/jul/17/elon-musk-regulation-ai-combat-existential-threat-tesla-spacex-ceo from the original on 6 June 2020. Retrieved 27 November 2017. ↑ cite ref-cnbc2 187-0 Kharpal, Arjun 7 November 2017 . "A.I. is in its 'infancy' and it's too early to regulate it, Intel CEO Brian Krzanich says" https://www.cnbc.com/2017/11/07/ai-infancy-and-too-early-to-regulate-intel-ceo-brian-krzanich-says.html . CNBC . Archived https://web.archive.org/web/20200322115325/https://www.cnbc.com/2017/11/07/ai-infancy-and-too-early-to-regulate-intel-ceo-brian-krzanich-says.html from the original on 22 March 2020. Retrieved 27 November 2017. ↑ cite ref-188 Dawes, James 20 December 2021 . "UN fails to agree on 'killer robot' ban as nations pour billions into autonomous weapons research" https://theconversation.com/un-fails-to-agree-on-killer-robot-ban-as-nations-pour-billions-into-autonomous-weapons-research-173616 . The Conversation . Retrieved 28 July 2023. 1 cite ref-:13 189-0 2 cite ref-:13 189-1 Fassihi, Farnaz 18 July 2023 . "U.N. Officials Urge Regulation of Artificial Intelligence" https://www.nytimes.com/2023/07/18/world/un-security-council-ai.html . The New York Times . ISSN https://en.wikipedia.org/wiki/ISSN identifier 0362-4331 https://search.worldcat.org/issn/0362-4331 . Retrieved 20 July 2023. ↑ cite ref-190 "International Community Must Urgently Confront New Reality of Generative, Artificial Intelligence, Speakers Stress as Security Council Debates Risks, Rewards" https://press.un.org/en/2023/sc15359.doc.htm . United Nations . Retrieved 20 July 2023. ↑ cite ref-191 Geist, Edward Moore 15 August 2016 . "It's already too late to stop the AI arms race—We must manage it instead". Bulletin of the Atomic Scientists . 72 5 : 318–321. Bibcode https://en.wikipedia.org/wiki/Bibcode identifier : 2016BuAtS..72e.318G https://ui.adsabs.harvard.edu/abs/2016BuAtS..72e.318G . doi https://en.wikipedia.org/wiki/Doi identifier : 10.1080/00963402.2016.1216672 https://doi.org/10.1080%2F00963402.2016.1216672 . ISSN https://en.wikipedia.org/wiki/ISSN identifier 0096-3402 https://search.worldcat.org/issn/0096-3402 . S2CID https://en.wikipedia.org/wiki/S2CID identifier 151967826 https://api.semanticscholar.org/CorpusID:151967826 . ↑ cite ref-192 "Amazon, Google, Meta, Microsoft and other tech firms agree to AI safeguards set by the White House" https://apnews.com/article/artificial-intelligence-safeguards-joe-biden-kamala-harris-4caf02b94275429f764b06840897436c . AP News . 21 July 2023. Retrieved 21 July 2023. ↑ cite ref-193 "Amazon, Google, Meta, Microsoft and other firms agree to AI safeguards" https://www.redditchadvertiser.co.uk/news/national/23670894.amazon-google-meta-microsoft-firms-agree-ai-safeguards/ . Redditch Advertiser . 21 July 2023. Retrieved 21 July 2023. ↑ cite ref-194 The White House 30 October 2023 . "Executive Order on the Safe, Secure, and Trustworthy Development and Use of Artificial Intelligence" https://bidenwhitehouse.archives.gov/briefing-room/presidential-actions/2023/10/30/executive-order-on-the-safe-secure-and-trustworthy-development-and-use-of-artificial-intelligence/ . The White House . Retrieved 19 December 2023. Bibliography edit /w/index.php?title=Existential risk from artificial intelligence&action=edit§ion=37 - Clark, Jack 2015a . "Musk-Backed Group Probes Risks Behind Artificial Intelligence" https://www.bloomberg.com/news/articles/2015-07-01/musk-backed-group-probes-risks-behind-artificial-intelligence . Bloomberg.com . Archived https://web.archive.org/web/20151030202356/http://www.bloomberg.com/news/articles/2015-07-01/musk-backed-group-probes-risks-behind-artificial-intelligence from the original on 30 October 2015. Retrieved 30 October 2015.