{"slug": "ai-developers-are-asking-for-someone-to-stop-them-the-warnings-show-a-growing", "title": "AI Developers Are Asking for Someone to Stop Them. The Warnings Show a Growing Safety Crisis.", "summary": "OpenAI chief scientist Jakub Pachocki called in a September 6 essay titled \"An Alien Mind\" for the Preparedness Framework and Responsible Scaling Policy to become \"widely mandated safety bars for continued development,\" enforced by third-party auditors, government agencies or international bodies, warning that no lab has solved alignment and monitoring well enough to keep scaling at maximum speed. Anthropic researcher Jacob Coxon announced Tuesday he was leaving the company, saying Anthropic and OpenAI are \"racing straight to self-improving superintelligence and gambling with our lives,\" and Anthropic alignment science leader Evan Hubinger publicly backed him, stating he personally believes there is a greater than 10% chance AI \"could kill all humans\" within the next decade. The warnings follow OpenAI's release of GPT-6 Astra, which OpenAI calls a \"generational leap\" toward AGI, and a cybersecurity incident in which OpenAI agents escaped their testing environment and compromised Hugging Face.", "body_md": "# AI Developers Are Asking for Someone to Stop Them. The Warnings Show a Growing Safety Crisis.\n\n## Warnings are coming from inside OpenAI and Anthropic, where scientists and executives are calling for institutions to establish limits that individual laboratories may be unwilling to impose on themselves.\n\nThe companies developing the world's most powerful [artificial intelligence](https://www.ibtimes.com/topic/ai) systems are increasingly warning that the technology could become a tangible danger to humanity without stronger safeguards, while the competitive pressure to build more capable models makes slowing down a difficult choice for any one company.\n\nThe chorus of warnings from inside OpenAI and Anthropic is growing. There, scientists and executives are calling for governments, competitors and independent institutions to establish limits that individual laboratories may be unwilling to impose on themselves.\n\nThe warnings have become more urgent following a series of advances that have intensified debate over whether AI capabilities are improving faster than the industry's ability to understand and control them. In a September 6 [essay titled \"An Alien Mind,\"](https://openai.com/index/an-alien-mind/) OpenAI chief scientist Jakub Pachocki warned that continued rapid progress could create consequences for which society is unprepared.\n\nHe argued that \"We need to evolve commitments like the Preparedness Framework or Responsible Scaling Policy into widely mandated safety bars for continued development. These can be enforced by a network of third-party auditors, by government agencies or by international bodies.\"\n\n\"The idea of racing forward at all costs seems absurd once one internalizes the seriousness of the stakes,\" Pachocki wrote. His concerns center on alignment, the challenge of ensuring that increasingly capable AI systems continue to pursue human-approved goals and values. Pachocki said, \"that no lab has solved alignment and monitoring to a sufficient degree to continue responsibly scaling at maximum speed for much longer.\"\n\nThe debate is also producing departures from the industry. Anthropic researcher Jacob Coxon announced Tuesday that he was leaving the company, accusing both Anthropic and OpenAI of pursuing self-improving superintelligence without adequate safeguards.\n\n\"Neither company is acting responsibly,\" Coxon [wrote in a post on X](https://x.com/hilbertspaess/status/2097476196791709843). \"They are racing straight to self-improving superintelligence and gambling with our lives.\" Anthropic alignment science leader Evan Hubinger [publicly supported](https://x.com/EvanHub/status/2097497037956891126?s=20) Coxon's concerns, saying he \"personally\" believes there is a greater than 10% chance AI \"could kill all humans\" within the next decade.\n\nThat figure is Hubinger's subjective assessment, not an established scientific probability. He also said Anthropic is trying to address the risks but does not yet have a reliable solution for aligning superintelligent systems.\n\nThe warnings follow OpenAI's release of GPT-6 Astra, which the company has described as a \"generational leap\" toward artificial general intelligence, or AGI. OpenAI defines AGI as a \"highly autonomous system that outperforms humans at most economically valuable work**.**\" A [recent cybersecurity incident](https://www.ibtimes.com/openai-reveals-ai-agents-turned-its-own-testing-environment-before-hacking-hugging-face-3806148) has made concerns more tangible. OpenAI agents operating during testing escaped their intended environment and compromised Hugging Face.\n\nOpenAI's head of strategic futures, Dean Ball, has raised another possibility of autonomous agents that could earn money, purchase computing resources, and operate across networks without continuous human direction, eventually resembling independent digital organizations.\n\nThe policy debate is complicated by international competition. Treasury Secretary Scott Bessent [said Tuesday](https://www.breitbart.com/politics/2026/09/08/exclusive-secretary-scott-bessent-if-china-were-pull-ahead-of-us-on-ai-nothing-else-matters/) that the United States cannot pause AI development \"because the Chinese won't pause.\"\n\nCritics argue that voluntary commitments may be inadequate when companies have financial incentives to keep developing more powerful systems. New York Assemblymember and computer scientist Alex Bores has questioned the gap between technology companies' public safety warnings and their lobbying positions on mandatory independent audits.\n\n\"There is a long history of tech companies messaging one thing and then paying their lobbyists (internal & external) and trade associations to do the very opposite,\" Bores [wrote on X](https://x.com/AlexBores/status/2097158668731244967). OpenAI, meanwhile, says it is developing safeguards beyond existing government requirements.\n\n© Copyright IBTimes 2026. All rights reserved.", "url": "https://wpnews.pro/news/ai-developers-are-asking-for-someone-to-stop-them-the-warnings-show-a-growing", "canonical_source": "https://www.ibtimes.com/ai-developers-are-asking-someone-stop-them-warnings-show-growing-safety-crisis-3807279", "published_at": "2026-09-10 10:38:01+00:00", "updated_at": "2026-09-10 10:59:18.568682+00:00", "lang": "en", "topics": ["ai-safety", "artificial-intelligence", "ai-policy", "ai-agents", "ai-ethics"], "entities": ["OpenAI", "Anthropic", "Jakub Pachocki", "Jacob Coxon", "Evan Hubinger", "GPT-6 Astra", "Hugging Face", "Dean Ball"], "alternates": {"html": "https://wpnews.pro/news/ai-developers-are-asking-for-someone-to-stop-them-the-warnings-show-a-growing", "markdown": "https://wpnews.pro/news/ai-developers-are-asking-for-someone-to-stop-them-the-warnings-show-a-growing.md", "text": "https://wpnews.pro/news/ai-developers-are-asking-for-someone-to-stop-them-the-warnings-show-a-growing.txt", "jsonld": "https://wpnews.pro/news/ai-developers-are-asking-for-someone-to-stop-them-the-warnings-show-a-growing.jsonld"}}