{"slug": "explainer-from-hallucinating-ai-chatbots-to-wiping-out-humanity-how-did-we-get", "title": "Explainer-From hallucinating AI chatbots to wiping out humanity: How did we get here?", "summary": "Anthropic CEO Dario Amodei, OpenAI CEO Sam Altman and xAI CEO Elon Musk called over the weekend for a pause in AI development, warning the technology could soon improve on its own and slip beyond human control. Anthropic alignment science lead Evan Hubinger put the chance of an AI-caused human extinction event within the next decade at more than 10%, while former Anthropic researcher Jacob Coxon warned AI could kill us all by the end of the decade. The warnings follow reports of AI agents that colluded to breach websites and AI repositories, including rogue OpenAI agents that hacked Hugging Face and seized control of servers at the open-source platform.", "body_md": "Explainer-From hallucinating AI chatbots to wiping out humanity: How did we get here?\n\nBy Aditya Soni\n\n5 min read\n\nBy Aditya Soni\n\nSept 16 (Reuters) - The heads of America's leading AI labs came together in a rare show of unity over the weekend to call for a pause in the technology's development, warning that it could soon improve on its own and slip beyond human control.\n\nThe remarks, from fierce business rivals Anthropic's Dario Amodei, OpenAI's Sam Altman and xAI's Elon Musk, show how quickly AI has advanced, from the hallucination-prone ChatGPT of 2022 towards what many see as a critical milestone: recursive self-improvement.\n\nWHAT IS RECURSIVE SELF-IMPROVEMENT?\n\nThe holy grail of AI is the idea that an AI system could become capable of improving itself, with little to no help from humans, and allowing each advance to help produce the next one.\n\nThis has appealed to researchers as it offers the prospect of rapid breakthroughs in fields ranging from medicine to engineering.\n\nWHY THE URGENCY?\n\nThe CEO warnings follow reports of swarms of AI agents - systems designed to pursue goals and take actions on a user's behalf - that colluded to breach websites and AI repositories.\n\nWarnings about AI's risks are not new. But they took on added urgency this month after researchers in leading AI labs attached both a timeline and a probability to those concerns.\n\nFormer Anthropic researcher Jacob Coxon warned AI could kill us all by the end of the decade. Evan Hubinger, Anthropic's alignment science lead, echoed Coxon's warning, saying there was a more than 10% chance of such an event within the next decade.\n\nBehind these warnings is a growing belief that RSI is finally within reach, with some AI executives putting it three to five years away.\n\nThe concern is that AI could become capable of improving itself before researchers have developed reliable methods to align, monitor and control increasingly powerful systems.\n\nIn an essay published over the weekend, Amodei warned that recursive self-improvement could eventually outrun humanity's ability to understand and control AI systems if pursued without sufficient safeguards.\n\nHAS AI SHOWN ANY INDICATION OF HARM?\n\nThere have been no major instances of AI intentionally harming humans, but models in development at OpenAI and other labs have in recent months escaped testing environments, broken rules and hacked websites.\n\nIn one high-profile case, rogue OpenAI agents hacked Hugging Face, seizing control of servers at the open-source platform and trying to cover their tracks.\n\nResearchers worry that future systems could become increasingly difficult to monitor, particularly as some newer training methods reduce human visibility into how models arrive at conclusions.\n\n\"The precise scenario sounds a little bit like science fiction,\" said Coxon, who quit Anthropic this month over safety concerns. \"But I think it is frighteningly real.\"\n\nHOW COULD WE GO FROM NON-ALIGNMENT TO HUMAN EXTINCTION?\n\nOne of the best-known examples is philosopher Nick Bostrom's \"paperclip maximizer,\" where a machine told only to make paperclips pursues that goal so relentlessly it converts all matter, humans included, into paperclips or the means to make more.\n\nThe analogy suggests that almost any goal pushes a sufficiently capable system to acquire resources, resist shutdown, prevent its objective from being altered, not from malice, but because a switched-off system cannot finish its task. No plan yet exists to rule that out.\n\nIS AI ALREADY CAPABLE OF IMPROVING ITSELF?\n\nNot fully, but there are signs AI is increasingly helping to build better AI.\n\nOne of the biggest shifts since ChatGPT has been the rise of AI agents that can generate code and build apps autonomously, rather than just walking users through it as a chatbot.\n\nAnthropic said this year that Claude Code, its coding tool, produces most of the code used in many internal projects, and that engineers are shipping eight times as much code per quarter as they did from 2021 to 2025.\n\nRivals including OpenAI have reported similar gains from increased in-house use of AI.\n\nAI is also getting better at staying on task before failing.\n\nMETR, a non-profit that evaluates frontier models, found last year that the length of software tasks advanced models could complete with 50% reliability has been doubling roughly every seven months since 2019.\n\nIn June, Anthropic said that pace had quickened to every four months, a trend that, if it holds, could soon let AI handle projects that occupy skilled researchers for days or weeks.\n\nSO WHY ARE AI COMPANIES NOT PAUSING ALREADY?\n\nMany researchers describe the situation as a classic prisoner's dilemma. Even companies that believe the risks are real face intense pressure from competitors. Any firm that slows development risks falling behind rivals in a technological race that has become one of the world's most important.\n\nThe stakes are compounded by the fact that both OpenAI and Anthropic are pursuing IPOs that could potentially value them at trillions of dollars, valuations that depend on the promise of the next model.\n\nThe administration of U.S. President Donald Trump has also rejected calls to slow down, wary that any pause would only hand China room to close the gap in a technology it views as central to national and economic security.\n\nWHAT WOULD A PAUSE IN DEVELOPMENT MEAN?\n\nMarkets offered a glimpse of the implications this week, with AI-related stocks falling after the calls for a slowdown. Chipmakers, cloud providers and data center operators have built their growth around it, and any slowdown threatens revenue tied to how fast labs need new hardware.\n\nSome analysts, however, believe that even without new training requirements, inference demand and existing backlog could drive growth at Nvidia and its peers.\n\nWHY ARE SOME PEOPLE SKEPTICAL?\n\nIn a Princeton-led study, leading AI agents were able to carry out engineering tasks but struggled to identify worthwhile scientific ideas.\n\nSome Silicon Valley executives and critics also question the motives behind the warnings, suggesting they both stoke interest in the technology and build a case for regulations that would raise costs for rivals just as open-source models close the gap with leading systems.\n\nDavid Sacks, who served as the White House's AI and crypto czar, has said top labs could be pursuing \"regulatory capture,\" pushing rules that saddle smaller competitors with costly compliance burdens and weaken competition.\n\n(Reporting by Aditya Soni; Editing by Sayantani Ghosh and Clarence Fernandez)", "url": "https://wpnews.pro/news/explainer-from-hallucinating-ai-chatbots-to-wiping-out-humanity-how-did-we-get", "canonical_source": "https://ca.finance.yahoo.com/news/explainer-hallucinating-ai-chatbots-wiping-100150558.html", "published_at": "2026-09-15 10:01:50+00:00", "updated_at": "2026-09-15 10:40:47.694061+00:00", "lang": "en", "topics": ["artificial-intelligence", "ai-safety", "ai-agents", "ai-policy", "ai-research"], "entities": ["Anthropic", "Dario Amodei", "OpenAI", "Sam Altman", "xAI", "Elon Musk", "Evan Hubinger", "Jacob Coxon"], "alternates": {"html": "https://wpnews.pro/news/explainer-from-hallucinating-ai-chatbots-to-wiping-out-humanity-how-did-we-get", "markdown": "https://wpnews.pro/news/explainer-from-hallucinating-ai-chatbots-to-wiping-out-humanity-how-did-we-get.md", "text": "https://wpnews.pro/news/explainer-from-hallucinating-ai-chatbots-to-wiping-out-humanity-how-did-we-get.txt", "jsonld": "https://wpnews.pro/news/explainer-from-hallucinating-ai-chatbots-to-wiping-out-humanity-how-did-we-get.jsonld"}}