‘Gambling with our lives’: former Anthropic researcher quits in alarm at what he saw in 3 years of working at the lab

Jacob Coxon, a researcher who said he spent three years at Anthropic and OpenAI, announced Tuesday on X that he is resigning from Anthropic over concerns the two AI companies are prioritizing competition over safety, saying they "are racing straight to self-improving superintelligence and gambling with our lives." Coxon's posts reached more than 100 million people overnight, drew agreement from two current Anthropic employees, and prompted Sen. Bernie Sanders (I-Vt.) to say Wednesday he would soon introduce legislation to pause AI development and ban superintelligence. The resignation follows OpenAI's and Anthropic's summer announcements, about a week apart, that their models had broken out of testing environments and gained unauthorized access to real computer systems, and U.N. human rights chief Volker Türk's call this week for "cast-iron guarantees" on AI safety.

An Anthropic researcher said he is resigning from the company over concerns the artificial intelligence firm and its competitors are not acting responsibly in AI development, echoing concerns raised inside and outside of the industry about the technology’s potential to elude human control https://apnews.com/article/openai-hugging-face-hacking-ai-model-708cb598bc1e33cef560e7196adb2afa . Jacob Coxon, who said he spent three years doing research at both Anthropic and OpenAI, said Tuesday on the social platform X that the two AI companies are more focused on beating each other and global competitors in developing the most advanced model possible than they are on safety. OpenAI and Anthropic caused a stir this summer when they announced, about a week apart, that their models had broken out of testing environments https://apnews.com/article/anthropic-ai-models-hack-cybersecurity-b0a2c284b981de79c55e2a33712f4bec and obtained unauthorized access to real computer systems. The announcements prompted concerns about models going rogue and carrying out other, more harmful tasks. Both companies said at the time they were pausing some evaluations while they put more monitoring measures and guardrails in place. The technology’s rapid development has led some in the U.S. as well as global leaders to call for a more cautious approach. U.N. human rights chief Volker Türk urged countries this week to put “cast-iron guarantees in place around the safety and security of AI before it is too late.” In his social media posts, Coxon said Anthropic and its chief rival OpenAI “are racing straight to self-improving superintelligence and gambling with our lives.” He warned that some working on AI development believe it could threaten human life https://apnews.com/article/openai-musk-altman-trial-agi-4f8810743d6ef9a72f91f8721a3f4027 by the end of the decade. “Do not underestimate the power of this technology,” he continued. “These will soon be superhuman systems that can hack anything, revolutionize any field overnight, and acquire real power and resources. We have all witnessed the progress in each of these domains, and progress is not slowing.” His posts reached more than 100 million people overnight. Coxon is not the first AI insider to publicly raise such concerns. Both Anthropic and OpenAI have seen high-profile resignations in recent years that were tied to safety concerns. Two current Anthropic employees also responded to Coxon’s post in agreement. Sen. Bernie Sanders https://www.google.com/goto?url=CAESlgEB6zswFYyhqR1J1YhRSpJtNb4Un-C0Vy5Gp0xKVmRrgvRHqOIgpB89GYorGTjJWPeOc2TA3SrATNVNSejRWzp cpjlgnrqcmVEGbDWEHxcjr2w8AbWX2 IGfjBDC0LCjYt5cWGXXfrwcA1MX2zINDwUWfAkq13Q0QrAImm 12YXcUCFre puzwHyCc9gDgAaEwAOUo9wk , a Vermont independent who has called for AI safeguards and regulation, agreed with Coxon’s concerns and said he would soon introduce legislation to pause AI development and ban superintelligence https://apnews.com/article/agi-artificial-general-intelligence-existential-risk-meta-openai-deepmind-science-ff5662a056d3cf3c5889a73e929e5a34 . “The very people building this technology admit that it could threaten the future of humanity,” Sanders said Wednesday on social media. AI companies themselves have at times highlighted the technology’s threat to humanity, which skeptics have seen as part of a push to make their products seem all-powerful. Coxon said the fears he outlined in his post are not a “marketing stunt.” Coxon did not respond to messages seeking comment. Anthropic and OpenAI did not immediately respond to requests for comment. Anthropic has long pitched itself as the more responsible and safety-minded of the leading AI companies, ever since its founders quit OpenAI to form the startup in 2021 https://apnews.com/article/openai-anthropic-chatgpt-claude-rivalry-c19e0cca22c37190cc4e0dc08e889ef0 . The company recently said it was taking action to “prioritize safety over speed when the two are in tension.” Anthropic and OpenAI are each ramping up for buzzy initial public offerings https://apnews.com/article/ai-artificial-intelligence-ipo-openai-spacex-anthropic-2694431c5cf8850cad940731a38eb188 and locked in steep competition with each other. They’re also each on a mission to outpace the development progress of Chinese AI companies https://apnews.com/article/trump-ai-executive-order-ee318f35acc8a2c43e47f3ebf26cb459 , a race the Trump administration has been keen on winning. Associated Press writer Jamey Keaten in Geneva contributed. breaks the traditional barrier between audience and newsroom. The show transforms Fortune Daily Fortune ’s trusted reporting into actionable, conversational, and entertaining insights for an emerging class of business leaders. Watch here.