{"slug": "ai-will-kill-us-all-unless-our-country-builds-it-first", "title": "AI Will Kill Us All, Unless Our Country Builds It First", "summary": "Former Anthropic researcher Jacob Coxon resigned and accused OpenAI and Anthropic of racing toward self-improving superintelligence while \"gambling with our lives,\" a claim Anthropic Alignment Science Lead Evan Hubinger publicly agreed with, estimating a greater than 10% probability of human extinction within ten years. The piece argues the resulting debate has become a contest over who gets to define \"AI safety,\" with Anthropic CEO Dario Amodei calling for slower global AI development and chip restrictions on China, Donald Trump dismissing extinction warnings as a hoax, and Chinese officials accusing American tech leaders of using safety as a pretext for containment. It contends that catastrophic AI risk stems from engineering and political choices—objectives, autonomy, resource access and the absence of a kill switch—rather than from a model magically crossing an intelligence threshold.", "body_md": "Former Anthropic researcher Jacob Coxon recently resigned and accused OpenAI and Anthropic of racing toward self-improving superintelligence while “gambling with our lives.” According to Coxon, many of the people building frontier AI genuinely believe it could kill all humans by the end of the decade.\n\nEvan Hubinger, Anthropic’s Alignment Science Lead, publicly agreed. His personal estimate was greater than a 10% probability of human extinction within the next ten years.\n\nThese were not journalists inventing an apocalyptic headline. These were people working close to the technology making explicit claims about extinction.\n\nThen the political machinery started.\n\nAnthropic CEO Dario Amodei called for the global development of AI to slow down. He also argued that a Chinese lead in AI would be a grave danger to the United States and the world, and that American restrictions on advanced chips supplied to China should continue.\n\nDonald Trump rejected the extinction warnings as a hoax. His position was simpler: the United States is leading China, “whoever wins AI, wins,” and slowing down would put America at a strategic disadvantage.\n\nChina responded by accusing American technology leaders of fearmongering and using safety as a pretext to contain Chinese development. At the same time, Chinese officials called for stronger systems to manage AI security risks, particularly when those risks involve political stability, ideology and control.\n\nApparently AI is dangerous when the other side is building it, safe when we are winning, and an existential threat when regulation reinforces our own position.\n\nThis is no longer a serious discussion about artificial intelligence. It is a competition for the authority to define what “AI safety” means.\n\nThe popular extinction story usually begins with an AI becoming more intelligent than humans. It escapes our control, takes over machines and infrastructure, eliminates us, and starts transforming the world according to its own plans.\n\nBut the story becomes strangely empty immediately after humanity disappears.\n\nWhat does the AI do next?\n\nDoes it create a civilization of robots? Why? Does it make Earth a better place? Better for whom? Does it clean the air, preserve forests and produce unlimited leisure for machines that neither breathe nor become tired?\n\n“Better” is not an objective property of the universe. It is a judgment made by something with needs, values and an experience of the world. Humans want clean air because our bodies require it. We want freedom because we experience constraint. We care about survival because death ends our experience.\n\nAn AI can represent all these concepts. It can discuss them convincingly because it learned from human language. That does not mean it automatically inherits the human condition behind them.\n\nWe imagine AI behaving like an empire because empire is a human pattern. We give it ambition, fear, greed, self-preservation and a desire to dominate. Then we become terrified of the human psychology we projected onto the machine.\n\nThis criticism does not prove that advanced AI is harmless. It proves that the cinematic explanation of the danger is weak.\n\nA system does not need to hate us to cause catastrophic damage. It only needs an objective incompatible with our interests, sufficient autonomy, access to critical resources, and no reliable mechanism for stopping it. Humans could become an obstacle without ever becoming an enemy.\n\nBut notice what is contained in that scenario: an objective, autonomy, access, resources and the absence of revocation.\n\nThose are engineering and political decisions. They are not magical properties acquired when a model crosses an intelligence threshold.\n\nNuclear energy is an obvious comparison because the same physical knowledge can power a city or erase one. Uranium has no opinion about either result. The bomb was not the inevitable destiny of nuclear physics. It was an application designed, funded, manufactured and deployed by humans.\n\nAI has a similar dual-use character. It can help discover medicines, make knowledge accessible and remove repetitive work. It can also scale surveillance, operate weapons, automate cyberattacks and manipulate populations.\n\nThe technology matters, but the deployment determines the consequence.\n\nYou can connect a model to a hospital or to a missile. You can let it recommend an action or execute it. You can give it read-only access or production credentials. You can place deterministic checks between its output and the real world, or call it an autonomous agent and celebrate that nobody needs to supervise it anymore.\n\nIf we build a system with broad permissions, opaque objectives, no independent verification and no reliable shutdown path, the surprising part is not that it becomes dangerous. The surprising part is that we connected it to anything important.\n\nThe nuclear analogy has limits. Nuclear capability depends on scarce materials, specialized facilities and visible supply chains. AI software can be copied and distributed at almost no marginal cost. It can also fail gradually inside ordinary products rather than through one recognizable catastrophic event.\n\nAI may therefore be harder to govern than nuclear technology in some respects. It is not, however, more supernatural.\n\nCivilization already consists largely of managing dangerous capabilities. Electricity, aviation, chemical engineering, medicine and nuclear power can all kill people. We did not solve those risks by insisting the underlying discoveries must never have happened. We created containment, testing, procedures, liability and institutions responsible for failure.\n\nCapability requires governance proportional to consequence.\n\nThe current AI debate is doing something different. It treats safety as a property of national ownership.\n\nAmodei’s position demonstrates the contradiction perfectly. Frontier AI may be so dangerous that global development must slow, but a Chinese lead would be uniquely dangerous, so the United States must preserve its advantage.\n\nTrump removes the first half and keeps the race. Extinction is a hoax because accepting it would justify slowing American development. The strategic conclusion has already been selected, so the risk analysis must conform to it.\n\nChina reverses the framing. American warnings are fearmongering when they justify chip controls and technological containment. AI risk becomes legitimate again when it threatens domestic political stability or creates dependence on foreign systems.\n\nEurope has its own version. It regulates risk while trying not to become permanently dependent on infrastructure and models owned elsewhere.\n\nEvery bloc claims to be protecting people. Every bloc defines protection in a way that preserves its own political and industrial interests.\n\nThis does not mean all AI safety research is propaganda. Alignment, containment, interpretability, cybersecurity and evaluation are real technical disciplines addressing real failures. Dismissing them would be as lazy as accepting every extinction forecast.\n\nThe political abuse begins when uncertain technical claims are converted into certainty only when certainty is useful.\n\nIf AI could plausibly kill everyone, then “we must win the race” is not a safety strategy. It is an admission that national advantage matters more than the stated existential danger. If AI cannot plausibly kill everyone, then using extinction to justify market barriers and concentrated control is regulatory capture dressed as responsibility.\n\nYou cannot coherently maintain that a technology is too dangerous for humanity to develop quickly while insisting that your own company or country must reach it first.\n\nThe contradiction disappears only if the real objective is control.\n\nI do not need a hypothetical superintelligence to identify a serious AI risk. We are already transferring cognitive infrastructure to a small number of companies because convenience wins one trivial decision at a time.\n\nThe idea for this article was already in my head. I could have spent two or three hours turning it into a structured argument. Instead, I used AI because expressing an opinion should not require sacrificing an afternoon.\n\nThe same calculation happens when we edit a photograph, summarize a document, translate a message, search through a codebase or write a function. Each request is insignificant. Collectively, those requests move an increasing share of intellectual work into systems we do not own, hosted in data centers we do not control, behind interfaces whose prices and behavior can change overnight.\n\nOrganizations remove human capacity because the automated path is cheaper. Skills weaken through disuse. Providers become infrastructure. Eventually the manual fallback exists only in a continuity document written by another AI.\n\nThat is not extinction. It is a concrete transfer of autonomy and power.\n\nThere is also a physical dependency underneath the interface. Generating a nicer profile picture consumes compute, electricity, cooling, chips and network capacity. One image does not destroy the planet, but billions of trivial requests becoming the standard interface for digital life create an infrastructure requirement that somebody must finance, operate and control.\n\nNo conscious machine needs to desire this outcome. Providers want adoption, markets reward convenience, and users prefer saving time. Ordinary incentives are sufficient.\n\nThis is how most dangerous systems actually emerge. Not through a machine announcing its intention to dominate humanity, but through thousands of locally reasonable decisions whose combined result nobody governs.\n\nWe need AI governance. But governance cannot mean preventing ordinary people from using models while allowing governments and the largest corporations to build anything they want behind closed doors.\n\nThat is not safety. It is technological feudalism.\n\nRegulation should follow capability, access and consequence. A local model helping somebody draft an article is not in the same risk category as an autonomous system authorized to move money, modify production infrastructure, operate weapons or launch code against external networks. Calling both “AI” does not make their danger equivalent.\n\nThe closer a system gets to consequential action, the stronger the evidence required before deployment should become. High-impact systems need defined failure modes, adversarial evaluation, restricted permissions, traceable decisions, independent monitoring, reversible actions and identifiable human operators.\n\n“The AI decided” must never become an acceptable technical explanation or legal defense.\n\nAt the same time, open research, interoperable tools and individual experimentation must remain possible. If safety rules make frontier capability accessible only to the same organizations lobbying for those rules, we have not protected society. We have protected incumbents.\n\nThere is no contradiction in supporting strict controls over AI connected to critical infrastructure while defending broad access to models that cannot directly produce high-impact actions. The relevant variable is not whether software is intelligent. It is what the system can do, what it can reach and who is accountable when it fails.\n\n“Whoever wins AI, wins” is an impressive concentration of bad assumptions in four words.\n\nAI is not a single finish line. There will be no final model that grants permanent ownership of intelligence to one nation. Capability is distributed across models, hardware, data, energy, talent and applications. Progress will continue to move between public research, private laboratories and open systems.\n\nMore importantly, a country winning does not mean its population wins. A national flag above a data center tells us nothing about who captures the economic value, who loses autonomy, who is subjected to automated decisions or who becomes responsible for the failures.\n\nHumanity does not benefit merely because the dominant system was trained on our side of a border.\n\nThe useful questions are much less theatrical. Who can inspect the system? Who can challenge its decisions? Who is liable? Can users leave? Can its permissions be revoked? Can a government or company secretly change its behavior? Does an independent fallback exist?\n\n“America versus China” answers none of them.\n\nWe should be capable of holding two ideas at the same time. AI can produce enormous harm. AI is not an autonomous demon with a natural desire to replace us.\n\nTreating it as harmless removes responsibility. Treating it as an inevitable machine apocalypse also removes responsibility. One says there is no danger. The other says the danger is destiny.\n\nThere is no destiny here.\n\nThere are models, objectives, infrastructure, permissions, incentives, institutions and people making decisions.\n\nIf AI is eventually used against humanity, it will not demonstrate that intelligence naturally becomes evil. It will demonstrate that we built a powerful instrument, connected it to systems that mattered, concentrated control over it, and then blamed the instrument for what followed.\n\nAI is not the country building it. It is not the company selling it. It is not the apocalypse being used to market regulation, and it is not the national victory being used to market acceleration.\n\nIt is a capability.\n\nThe danger is who controls it, what they connect it to, and whether the rest of us are still allowed to say no.", "url": "https://wpnews.pro/news/ai-will-kill-us-all-unless-our-country-builds-it-first", "canonical_source": "https://dev.to/marcosomma/ai-will-kill-us-all-unless-our-country-builds-it-first-2mpe", "published_at": "2026-09-16 08:00:51+00:00", "updated_at": "2026-09-16 08:13:27.882616+00:00", "lang": "en", "topics": ["ai-safety", "ai-policy", "ai-ethics", "artificial-intelligence"], "entities": ["Anthropic", "OpenAI", "Jacob Coxon", "Evan Hubinger", "Dario Amodei", "Donald Trump", "China", "United States"], "alternates": {"html": "https://wpnews.pro/news/ai-will-kill-us-all-unless-our-country-builds-it-first", "markdown": "https://wpnews.pro/news/ai-will-kill-us-all-unless-our-country-builds-it-first.md", "text": "https://wpnews.pro/news/ai-will-kill-us-all-unless-our-country-builds-it-first.txt", "jsonld": "https://wpnews.pro/news/ai-will-kill-us-all-unless-our-country-builds-it-first.jsonld"}}