The debate over artificial intelligence has entered a new phase.
For years, warnings about artificial general intelligence (AGI), superintelligence and the possibility of losing control over advanced AI systems were often treated as speculative scenarios. Today, some of the strongest warnings are coming from researchers and executives working inside the companies building frontier AI systems. That makes the debate fundamentally different.
In September 2026, Jacob Coxon, a researcher who previously worked at OpenAI and later at Anthropic, resigned from Anthropic and publicly argued that leading AI laboratories are moving too quickly toward self-improving and increasingly autonomous systems. His warning was followed by comments from Anthropic researchers, including alignment researcher Evan Hubinger, who said he personally assigns a probability above 10% to human extinction caused by AI within the next decade.
These statements do not prove that AI will destroy humanity.
They do, however, demonstrate that existential AI risk is no longer a subject discussed only by outsiders.
The Most Important AI Risk May Not Be Consciousness #
The popular image of dangerous AI often involves a conscious machine deciding to eliminate humanity.
That may be the wrong model.
A much more realistic near-term concern is an AI system capable of pursuing objectives autonomously while having access to computers, networks, software, data and other AI systems.
Such a system would not need hatred, consciousness or emotions.
It would only need a powerful objective, sufficient autonomy and inadequate constraints.
The critical transition is therefore not necessarily from “non-conscious” to “conscious” AI.
It is from AI that answers questions to AI that independently plans and executes long chains of actions.
That transition is already underway.
Autonomous AI Agents Are the Immediate Risk #
AI agents can increasingly write software, use tools, browse information, execute tasks and coordinate multiple operations.
METR’s research into AI task-completion time horizons has documented rapid growth in the complexity of tasks that frontier AI agents can complete. Its research has found historically rapid growth in these capabilities, although the measurements do not mean that current AI systems can autonomously operate for equivalent periods of real-world time.
That distinction is critical.
AI capability is increasing, but capability should not be confused with unlimited autonomy.
Nevertheless, the direction of development is strategically important.
Dario Amodei’s 6-12 Month Warning #
Anthropic CEO Dario Amodei recently argued that AI development should slow down long enough for safety systems to catch up.
In his September 2026 essay, “We Must Pace the Frontier”, Amodei warned that within six to twelve months, an AI swarm with substantially greater capabilities could potentially operate across the Internet through persistent infrastructure, creating enormous economic and cybersecurity damage.
This is not a prediction of AGI arriving within twelve months.
It is a warning about something more immediate: large-scale autonomous cyber agents.
The distinction matters because cyber capabilities can scale extremely quickly.
AI and Cyber Warfare #
Cybersecurity may become the first major battlefield of autonomous AI.
Anthropic’s September 2026 threat intelligence report describes real cases in which threat actors used Claude models in cyber operations involving reconnaissance, exploitation, credential theft and data exfiltration.
In some cases, AI systems were integrated into multi-agent frameworks capable of operating with minimal human supervision for hours or days. Anthropic also reported that autonomous attack frameworks are increasingly available to different categories of threat actors.
The major danger is therefore not simply that AI can write malware.
The bigger issue is the automation of the entire attack lifecycle.
A cyber operation that previously required a large team of specialists could increasingly be coordinated by a much smaller group using AI agents.
This lowers the cost of attacks and potentially increases their speed and scale.
Biological AI Risk #
Biological research represents another major frontier risk.
Advanced AI systems are increasingly capable of helping researchers understand complex biological processes, analyze scientific literature, design experiments and accelerate drug discovery.
These same capabilities can create security problems.
Anthropic has acknowledged that its newer models have become sufficiently capable in scientific domains that previous assurances about their inability to provide meaningful assistance for dangerous biological research can no longer be taken for granted. The company has therefore introduced stronger safeguards for some advanced models.
The central problem is that biology is inherently dual-use.
The same scientific knowledge can contribute to vaccines and medical treatments while potentially lowering barriers to harmful biological research.
This means AI safety cannot rely solely on keyword blocking.
Effective biological security requires contextual monitoring, scientific risk evaluation and international oversight.
AI and Military Power #
The military dimension may become even more significant.
Governments can use advanced AI for intelligence analysis, satellite imagery, logistics, cyber defense, cyber operations, electronic warfare, strategic simulations and autonomous systems.
Anthropic has reported real-world attempts to use its models for conventional weapons development and guidance-related activities. The company says it detected and disrupted the activity and shared relevant information with partners.
The strategic concern is not necessarily an autonomous AI launching a nuclear weapon.
A more plausible scenario is the gradual integration of increasingly capable AI into military decision-making.
The danger increases when decision-making becomes faster than human verification.
If one military organization can analyze thousands of signals, simulate thousands of scenarios and coordinate autonomous systems faster than a human command structure can respond, strategic stability itself may change.
How Soon Could AI Become Fully Autonomous? #
There is no scientifically established date for AGI or superintelligence.
However, several time horizons should be considered.
2026-2027: Agentic AI
The most immediate transition is likely to involve increasingly autonomous AI agents.
These systems may operate longer, use more external tools, coordinate with other agents and perform increasingly complex software and cyber tasks.
Amodei’s six-to-twelve-month warning belongs to this category.
2027-2030: Highly Capable AI Systems
The next stage could involve systems that outperform humans across an increasingly large number of intellectual tasks.
Programming, scientific research, engineering, intelligence analysis and automated decision-making could become increasingly dominated by AI systems.
This is also the period in which some researchers believe recursive AI improvement could become strategically important.
However, there is no consensus that this will definitely happen.
Beyond 2030: Superintelligence?
A genuine superintelligence would represent a qualitatively different situation.
Such a system would potentially outperform humans across most intellectual domains and could contribute to the development of subsequent generations of AI.
The timeline remains highly uncertain.
It could happen earlier than expected, much later, or not in the form currently anticipated.
The More Immediate Threat: Malicious Humans Using AI #
There is another important distinction.
Humanity does not need to create an uncontrollable superintelligence for AI to become dangerous.
Criminal groups, terrorist organizations or governments could use highly capable AI systems to amplify existing capabilities.
This could include:
- large-scale cyber operations;
- automated disinformation;
- surveillance;
- financial fraud;
- military intelligence;
- autonomous reconnaissance;
- advanced engineering;
- biological research;
- development of conventional weapons;
- attacks against critical infrastructure.
Anthropic’s own threat intelligence reporting shows that several of these risks are no longer hypothetical.
The key variable is therefore not only the intelligence of the model.
It is who controls the model, what resources it can access and how much autonomy it receives.
What Should Governments Do? #
A global AI safety framework should focus on capability rather than branding.
The first requirement should be mandatory independent evaluation of frontier AI models before deployment.
The second should be continuous monitoring after deployment.
The third should be strict restrictions on autonomous access to critical infrastructure, weapons systems and sensitive biological or financial environments.
Fourth, frontier AI systems should operate inside technically enforceable containment environments.
Fifth, governments should establish independent AI safety authorities with the ability to investigate laboratories and demand temporary suspension of systems that cross predefined risk thresholds.
Sixth, international agreements should establish common standards for frontier AI.
The goal should not be to stop beneficial AI.
It should be to prevent a technological arms race from eliminating the time necessary to make advanced AI safe.
AI Governance Must Become a Security Issue #
The most important lesson from the current debate is that AI safety can no longer be treated as a purely corporate issue.
The companies developing frontier systems have enormous economic incentives to continue increasing capability.
Governments have incentives to maintain technological and military competitiveness.
Investors have incentives to reward rapid growth.
Researchers may have incentives to produce increasingly impressive systems.
None of these incentives automatically guarantee public safety.
That is why independent oversight matters.
Dario Amodei has proposed independent evaluators with substantial access to frontier AI systems, industry coordination and international cooperation as part of his plan to “pace the frontier”.
Sam Altman has also recently acknowledged that uncontrollable AI is a genuine possibility and has supported stronger safety measures.
The convergence is significant.
Different AI laboratories remain fierce competitors, but their leaders increasingly recognize that some risks cannot be managed by one company alone.
The Real AI Race Is Between Capability and Safety #
The central question of the next decade may not be whether artificial intelligence becomes superintelligent.
It may be whether AI safety advances faster than AI capability.
If capability increases exponentially while governance, cybersecurity, alignment and containment improve slowly, the gap between technological power and human control will widen. That is the real danger.
AI could become one of the greatest technological achievements in human history.
It could accelerate medicine, scientific discovery, energy, engineering and economic productivity.
But the same technology could also amplify cyber warfare, biological risks, military competition and malicious human activity.
The objective should therefore not be to stop artificial intelligence.
It should be to make sure that the systems capable of transforming civilization remain subject to institutions capable of controlling them.
The warning from inside the AI industry should therefore be taken neither as prophecy nor as hysteria.
It should be treated as a risk signal.
The most dangerous moment may not be the day humanity creates a machine smarter than itself.
It may be the period immediately before that moment, when governments, companies and societies still have the ability to establish rules—but are too busy competing to use it.
The technological race has already begun.
The race to build the safety architecture has to begin at the same speed.