Could AI really kill us all – and how can we stop it?

Former Anthropic and OpenAI researcher Jacob Coxon said in a viral post and CNN interview that his former employers are "racing straight to self-improving superintelligence" that could "kill us all by the end of the decade," a claim that drew responses from tech bosses, regulators and world leaders. Vu Tran, who previously worked at Meta Superintelligence Labs, wrote on X that "we are already past the point where AI can destroy the world," while Coxon said current systems pose no extinction risk but warned about the rate of progress. Anthropic CEO Dario Amodei argued in a Saturday essay that AI firms should "slow the pace at which we improve the capabilities of AI models.

Could AI really kill us all – and how can we stop it? One researcher says ‘we are already past the point where AI can destroy the world’. Anthony Cuthbertson https://www.independent.co.uk/author/anthony-cuthbertson looks at how it might happen, and if we should be worried - Bookmark When Jacob Coxon quit his job at Anthropic last week, he did not originally plan to share his reasons for doing so. But after chatting with some friends, he decided to make a post https://www.independent.co.uk/tech/anthropic-openai-ai-threat-jacob-coxon-b3047033.html detailing why he could no longer be part of an industry that is “gambling with our lives”. By the weekend, his post had been viewed more than 100 million times. “I did not expect it to go this viral,” he said on an interview with CNN. “But clearly there was latent demand for this thing to absolutely explode.” The fall-out from his post has drawn responses from tech bosses, regulators and even world leaders. So how worried should we all be? Could AI really kill us? According to Coxon, who has also previously worked at OpenAI, his former employers are “racing straight to self-improving superintelligence” that could “kill us all by the end of the decade”. Superintelligence https://www.independent.co.uk/tech/ai-superintelligence-openai-chatgpt-extinction-b2348800.html , which refers to artificial intelligence that exceeds the cognitive abilities of humans in all domains”, is still a long way off, but some believe AI has already reached parity with humans. Following the release of OpenAI’s GPT-6 model earlier this month, Nvidia CEO Jensen Huang declared that it marked the arrival of human-level AI https://www.independent.co.uk/tech/chatgpt-astra-agi-nvidia-huang-ai-b3046386.html , known as artificial general intelligence AGI . According to one researcher, we are also already past the point that AI could wipe out humanity, albeit with human assistance. Vu Tran, who previously worked at Meta Superintelligence Labs, claimed last week that leading AI firms already have powerful enough systems to wreak havoc. “If OpenAI wanted to cripple an entire nation, they easily could today,” he wrote on X. “All they’d have to do is remove alignment and unleash an agent swarm. It could probably within a day or so get access to all of the nation’s data centres and shut off all the country’s utilities. “Like, we are already past the point where AI can destroy the world. Do people realise this?” How would it happen? There are various hypothetical scenarios that researchers have warned about when it comes to the existential threat posed by advanced artificial intelligence. When asked by Anderson Cooper on CNN how he thought AI could kill all humans, Jacob Coxon referred to an incident that happened in July, when one of OpenAI’s experimental models launched a cyber attack on the AI platform Hugging Face without being instructed to. “I think that if you extrapolate into the future the level of capabilities of these AIs, with the same independent volition, they could cause extreme havoc,” he said. “For example, hacking critical infrastructure, building extinction-level bioweapons, there’s a lot of ways that the AI could actuate itself in the world.” For now the threat is purely hypothetical, according to Coxon, who claims that current technologies are not powerful enough to carry out such world-ending tasks. But the current development trajectory of frontier models means we are not far away. “Right now, there’s no risk of extinction,” he said. “What’s just crazy is to look at the rate of progress.” How do we stop it? It is widely acknowledged among leading industry figures that AI safety protocols need time to catch up with the capabilities of frontier models, but no one is willing to hit pause on their development unless everyone else does. This requires international consensus and some form of universally accepted regulation. In an essay published on Saturday, Anthropic CEO Dario Amodei argued that AI firms should “slow the pace at which we improve the capabilities of AI models”, or risk a major catastrophe in the very near future. “Given the accelerating rate of AI capability development, it’s my worry that in 6–12 months an AI swarm could be capable of taking over the entire internet with a persistent botnet,” he wrote. “The scale of damage would continue to increase from there if AI becomes more powerful without the necessary guardrails.” His three-part plan https://darioamodei.com/post/we-must-pace-the-frontier received the backing of OpenAI CEO Sam Altman, Google DeepMind’s Demis Hassabis and SpaceXAI boss Elon Musk, though the rare show of unity among the rival tech leaders was labeled a “SICK conspiracy” by US President Donald Trump. Any new rules, Trump claimed, would see the US fall behind China in the AI race. Outside of the US, the most powerful AI models are currently being developed by Chinese companies. Frontier labs like DeepSeek and Manus, as well as tech giants like Alibaba and Baidu, have been building powerful systems with the backing of Beijing, though they still lag behind the leading US firms in key benchmark tests. Writing on Truth Social, Trump suggested that he could single-handedly keep the industry in check and save the world from an AI-induced disaster. “The only control or ‘guardrails’ that AI needs is a STRONG AND SMART High IQ PRESIDENT, and the U.S.A. has that, in spades ” he wrote https://www.independent.co.uk/news/world/americas/us-politics/trump-ai-rant-conspiracy-guardrails-b3049920.html . Without regulatory reform, one option would be for companies to build a kill switch into their most powerful models in order to stop a dangerous attack. This idea was proposed by Google DeepMind in 2016, in a peer-reviewed paper https://www.newsweek.com/google-big-red-button-ai-artificial-intelligence-save-world-elon-musk-466753 titled ‘Safely interruptible agents’. The researchers outlined a framework for preventing advanced AI from ignoring turn-off commands through a “big red button” that could switch off any rogue AI. There have since been several efforts to adopt this type of technology, most recently from legislation put forward by lords and MPs in the UK. Led by the Liberal Democrats’ Lord Tim Clement-Jones, the amendment to the Cyber Security and Resilience Bill would offer a “vital safety net” to “halt a runaway system before it can compromise our critical national infrastructure”. The proposal was rejected last week by the Cabinet Office, who claimed that blocking models in the UK would not prevent them from being misused elsewhere. Join our commenting forum Join thought-provoking conversations, follow other Independent readers and see their replies Comments comments-area