{"slug": "ai-existential-risk-how-close-are-we-to-autonomous-ai-superintelligence-and-ai", "title": "AI Existential Risk: How Close Are We to Autonomous AI, Superintelligence and AI-Powered Warfare?", "summary": "Anthropic alignment researcher Evan Hubinger said he assigns a probability above 10% to human extinction caused by AI within the next decade, following the September 2026 resignation of Jacob Coxon, a researcher who left Anthropic after previously working at OpenAI and publicly argued that leading AI laboratories are moving too quickly toward self-improving and increasingly autonomous systems. Anthropic CEO Dario Amodei warned in his September 2026 essay \"We Must Pace the Frontier\" that within six to twelve months an AI swarm with substantially greater capabilities could operate across the Internet through persistent infrastructure, creating enormous economic and cybersecurity damage. Anthropic's September 2026 threat intelligence report described real cases in which threat actors used Claude models in cyber operations involving reconnaissance, exploitation, credential theft and data exfiltration, with some AI systems integrated into multi-agent frameworks operating with minimal human supervision for hours or days.", "body_md": "# AI Existential Risk: How Close Are We to Autonomous AI, Superintelligence and AI-Powered Warfare?\n\nThe debate over artificial intelligence has entered a new phase.\n\nFor years, warnings about artificial general intelligence (AGI), superintelligence and the possibility of losing control over advanced AI systems were often treated as speculative scenarios. Today, some of the strongest warnings are coming from researchers and executives working inside the companies building frontier AI systems.\n\nThat makes the debate fundamentally different.\n\nIn September 2026, Jacob Coxon, a researcher who previously worked at OpenAI and later at Anthropic, resigned from Anthropic and publicly argued that leading AI laboratories are moving too quickly toward self-improving and increasingly autonomous systems. His warning was followed by comments from Anthropic researchers, including alignment researcher Evan Hubinger, who said he personally assigns a probability above 10% to human extinction caused by AI within the next decade.\n\nThese statements do not prove that AI will destroy humanity.\n\nThey do, however, demonstrate that existential AI risk is no longer a subject discussed only by outsiders.\n\n## The Most Important AI Risk May Not Be Consciousness\n\nThe popular image of dangerous AI often involves a conscious machine deciding to eliminate humanity.\n\nThat may be the wrong model.\n\nA much more realistic near-term concern is an AI system capable of pursuing objectives autonomously while having access to computers, networks, software, data and other AI systems.\n\nSuch a system would not need hatred, consciousness or emotions.\n\nIt would only need a powerful objective, sufficient autonomy and inadequate constraints.\n\nThe critical transition is therefore not necessarily from “non-conscious” to “conscious” AI.\n\nIt is from **AI that answers questions to AI that independently plans and executes long chains of actions**.\n\nThat transition is already underway.\n\n## Autonomous AI Agents Are the Immediate Risk\n\nAI agents can increasingly write software, use tools, browse information, execute tasks and coordinate multiple operations.\n\nMETR’s research into AI task-completion time horizons has documented rapid growth in the complexity of tasks that frontier AI agents can complete. Its research has found historically rapid growth in these capabilities, although the measurements do not mean that current AI systems can autonomously operate for equivalent periods of real-world time.\n\nThat distinction is critical.\n\nAI capability is increasing, but capability should not be confused with unlimited autonomy.\n\nNevertheless, the direction of development is strategically important.\n\n## Dario Amodei’s 6-12 Month Warning\n\nAnthropic CEO Dario Amodei recently argued that AI development should slow down long enough for safety systems to catch up.\n\nIn his September 2026 essay, “We Must Pace the Frontier”, Amodei warned that within six to twelve months, an AI swarm with substantially greater capabilities could potentially operate across the Internet through persistent infrastructure, creating enormous economic and cybersecurity damage.\n\nThis is not a prediction of AGI arriving within twelve months.\n\nIt is a warning about something more immediate: **large-scale autonomous cyber agents**.\n\nThe distinction matters because cyber capabilities can scale extremely quickly.\n\n## AI and Cyber Warfare\n\nCybersecurity may become the first major battlefield of autonomous AI.\n\nAnthropic’s September 2026 threat intelligence report describes real cases in which threat actors used Claude models in cyber operations involving reconnaissance, exploitation, credential theft and data exfiltration.\n\nIn some cases, AI systems were integrated into multi-agent frameworks capable of operating with minimal human supervision for hours or days. Anthropic also reported that autonomous attack frameworks are increasingly available to different categories of threat actors.\n\nThe major danger is therefore not simply that AI can write malware.\n\nThe bigger issue is the **automation of the entire attack lifecycle**.\n\nA cyber operation that previously required a large team of specialists could increasingly be coordinated by a much smaller group using AI agents.\n\nThis lowers the cost of attacks and potentially increases their speed and scale.\n\n## Biological AI Risk\n\nBiological research represents another major frontier risk.\n\nAdvanced AI systems are increasingly capable of helping researchers understand complex biological processes, analyze scientific literature, design experiments and accelerate drug discovery.\n\nThese same capabilities can create security problems.\n\nAnthropic has acknowledged that its newer models have become sufficiently capable in scientific domains that previous assurances about their inability to provide meaningful assistance for dangerous biological research can no longer be taken for granted. The company has therefore introduced stronger safeguards for some advanced models.\n\nThe central problem is that biology is inherently dual-use.\n\nThe same scientific knowledge can contribute to vaccines and medical treatments while potentially lowering barriers to harmful biological research.\n\nThis means AI safety cannot rely solely on keyword blocking.\n\nEffective biological security requires contextual monitoring, scientific risk evaluation and international oversight.\n\n## AI and Military Power\n\nThe military dimension may become even more significant.\n\nGovernments can use advanced AI for intelligence analysis, satellite imagery, logistics, cyber defense, cyber operations, electronic warfare, strategic simulations and autonomous systems.\n\nAnthropic has reported real-world attempts to use its models for conventional weapons development and guidance-related activities. The company says it detected and disrupted the activity and shared relevant information with partners.\n\nThe strategic concern is not necessarily an autonomous AI launching a nuclear weapon.\n\nA more plausible scenario is the gradual integration of increasingly capable AI into military decision-making.\n\nThe danger increases when decision-making becomes faster than human verification.\n\nIf one military organization can analyze thousands of signals, simulate thousands of scenarios and coordinate autonomous systems faster than a human command structure can respond, strategic stability itself may change.\n\n## How Soon Could AI Become Fully Autonomous?\n\nThere is no scientifically established date for AGI or superintelligence.\n\nHowever, several time horizons should be considered.\n\n### 2026-2027: Agentic AI\n\nThe most immediate transition is likely to involve increasingly autonomous AI agents.\n\nThese systems may operate longer, use more external tools, coordinate with other agents and perform increasingly complex software and cyber tasks.\n\nAmodei’s six-to-twelve-month warning belongs to this category.\n\n### 2027-2030: Highly Capable AI Systems\n\nThe next stage could involve systems that outperform humans across an increasingly large number of intellectual tasks.\n\nProgramming, scientific research, engineering, intelligence analysis and automated decision-making could become increasingly dominated by AI systems.\n\nThis is also the period in which some researchers believe recursive AI improvement could become strategically important.\n\nHowever, there is no consensus that this will definitely happen.\n\n### Beyond 2030: Superintelligence?\n\nA genuine superintelligence would represent a qualitatively different situation.\n\nSuch a system would potentially outperform humans across most intellectual domains and could contribute to the development of subsequent generations of AI.\n\nThe timeline remains highly uncertain.\n\nIt could happen earlier than expected, much later, or not in the form currently anticipated.\n\n## The More Immediate Threat: Malicious Humans Using AI\n\nThere is another important distinction.\n\nHumanity does not need to create an uncontrollable superintelligence for AI to become dangerous.\n\nCriminal groups, terrorist organizations or governments could use highly capable AI systems to amplify existing capabilities.\n\nThis could include:\n\n- large-scale cyber operations;\n- automated disinformation;\n- surveillance;\n- financial fraud;\n- military intelligence;\n- autonomous reconnaissance;\n- advanced engineering;\n- biological research;\n- development of conventional weapons;\n- attacks against critical infrastructure.\n\nAnthropic’s own threat intelligence reporting shows that several of these risks are no longer hypothetical.\n\nThe key variable is therefore not only the intelligence of the model.\n\nIt is **who controls the model, what resources it can access and how much autonomy it receives**.\n\n## What Should Governments Do?\n\nA global AI safety framework should focus on capability rather than branding.\n\nThe first requirement should be mandatory independent evaluation of frontier AI models before deployment.\n\nThe second should be continuous monitoring after deployment.\n\nThe third should be strict restrictions on autonomous access to critical infrastructure, weapons systems and sensitive biological or financial environments.\n\nFourth, frontier AI systems should operate inside technically enforceable containment environments.\n\nFifth, governments should establish independent AI safety authorities with the ability to investigate laboratories and demand temporary suspension of systems that cross predefined risk thresholds.\n\nSixth, international agreements should establish common standards for frontier AI.\n\nThe goal should not be to stop beneficial AI.\n\nIt should be to prevent a technological arms race from eliminating the time necessary to make advanced AI safe.\n\n## AI Governance Must Become a Security Issue\n\nThe most important lesson from the current debate is that AI safety can no longer be treated as a purely corporate issue.\n\nThe companies developing frontier systems have enormous economic incentives to continue increasing capability.\n\nGovernments have incentives to maintain technological and military competitiveness.\n\nInvestors have incentives to reward rapid growth.\n\nResearchers may have incentives to produce increasingly impressive systems.\n\nNone of these incentives automatically guarantee public safety.\n\nThat is why independent oversight matters.\n\nDario Amodei has proposed independent evaluators with substantial access to frontier AI systems, industry coordination and international cooperation as part of his plan to “pace the frontier”.\n\nSam Altman has also recently acknowledged that uncontrollable AI is a genuine possibility and has supported stronger safety measures.\n\nThe convergence is significant.\n\nDifferent AI laboratories remain fierce competitors, but their leaders increasingly recognize that some risks cannot be managed by one company alone.\n\n## The Real AI Race Is Between Capability and Safety\n\nThe central question of the next decade may not be whether artificial intelligence becomes superintelligent.\n\nIt may be whether **AI safety advances faster than AI capability**.\n\nIf capability increases exponentially while governance, cybersecurity, alignment and containment improve slowly, the gap between technological power and human control will widen.\n\nThat is the real danger.\n\nAI could become one of the greatest technological achievements in human history.\n\nIt could accelerate medicine, scientific discovery, energy, engineering and economic productivity.\n\nBut the same technology could also amplify cyber warfare, biological risks, military competition and malicious human activity.\n\nThe objective should therefore not be to stop artificial intelligence.\n\nIt should be to make sure that the systems capable of transforming civilization remain subject to institutions capable of controlling them.\n\nThe warning from inside the AI industry should therefore be taken neither as prophecy nor as hysteria.\n\nIt should be treated as a **risk signal**.\n\nThe most dangerous moment may not be the day humanity creates a machine smarter than itself.\n\nIt may be the period immediately before that moment, when governments, companies and societies still have the ability to establish rules—but are too busy competing to use it.\n\nThe technological race has already begun.\n\nThe race to build the safety architecture has to begin at the same speed.", "url": "https://wpnews.pro/news/ai-existential-risk-how-close-are-we-to-autonomous-ai-superintelligence-and-ai", "canonical_source": "https://www.quantumhorizon.it/ai-existential-risk/", "published_at": "2026-09-13 16:04:45+00:00", "updated_at": "2026-09-13 16:14:14.833681+00:00", "lang": "en", "topics": ["ai-safety", "ai-agents", "artificial-intelligence", "ai-policy", "ai-ethics"], "entities": ["Anthropic", "OpenAI", "Jacob Coxon", "Evan Hubinger", "Dario Amodei", "Claude", "METR", "We Must Pace the Frontier"], "alternates": {"html": "https://wpnews.pro/news/ai-existential-risk-how-close-are-we-to-autonomous-ai-superintelligence-and-ai", "markdown": "https://wpnews.pro/news/ai-existential-risk-how-close-are-we-to-autonomous-ai-superintelligence-and-ai.md", "text": "https://wpnews.pro/news/ai-existential-risk-how-close-are-we-to-autonomous-ai-superintelligence-and-ai.txt", "jsonld": "https://wpnews.pro/news/ai-existential-risk-how-close-are-we-to-autonomous-ai-superintelligence-and-ai.jsonld"}}