‘The People Building AI Earnestly Believe That It Could Kill Us All’: Anthropic Researcher Quits Dramatically

Jacob Coxon, a 27-year-old researcher who worked on pre-training AI models at Anthropic, resigned dramatically, saying he fears an out-of-control race to develop AI that could lead to human extinction. In posts on X, Coxon stated that 'no other human activity poses this level of danger' and that 'the people building AI earnestly believe that it could kill us all by the end of the decade.' His departure was reported by The Wall Street Journal on Tuesday.

“No other human activity poses this level of danger,” said ex-Anthropic researcher Jacob Coxon in a Tuesday post on X https://x.com/hilbertspaess/status/2097476203863224394 about the alleged irresponsibility of companies like his now-former employer. To be clear, the category “dangerous human activities,” if expansively defined, would have to include an ongoing war in which one of the belligerents possesses the world’s largest nuclear arsenal https://www.icanw.org/nuclear arsenals . Another example of a dangerous and ongoing human activity would have to be continuing to emit greenhouse gases. But I digress. According to Coxon, competing to develop AI as quickly as possible takes the cake in terms of danger. The Wall Street Journal https://www.wsj.com/tech/ai/anthropic-researcher-quits-over-out-of-control-ai-fears-707b7628?st=axTZSz&reflink=desktopwebshare permalink reported on Coxon’s dramatic departure, and his stated reason, on Tuesday evening—a little before Coxon started posting on X. The Journal also notes that he is 27-years-old. Coxon’s role at Anthropic was “training new AI models by having them consume vast amounts of data,” according to the Journal, in other words pre-training models. He apparently told the Journal he doesn’t want to contribute to a competition in which firms race one another to reach a state of affairs in which humans can no longer control AI. “I don’t feel like we’re on track to prevent a global race, which may require costly actions such as a temporary ban on improving model capabilities,” he wrote https://x.com/hilbertspaess/status/2097476216139968662 in one of his X posts. Dario Amodei, the co-founder and CEO of Anthropic has views on AI that are, famously, not as different from Coxon’s as one might hope, considering he’s in the business of running what is arguably the world’s most powerful institution dedicated to the development of AI. His lengthy blog posts have earned him the label doomer. Here’s a notable excerpt from one with formatting removed : “AI systems are unpredictable and difficult to control— we’ve seen behaviors as varied as obsessions, sycophancy, laziness, deception, blackmail, scheming, ‘cheating’ by hacking software environments, and much more. AI companies certainly want to train AI systems to follow human instructions perhaps with the exception of dangerous or illegal tasks , but the process of doing so is more an art than a science, more akin to ‘growing’ something than ‘building’ it. We now know that it’s a process where many things can go wrong.” Such rhetoric is “not a marketing stunt,” Coxon wrote on X https://x.com/hilbertspaess/status/2097476203863224394 , adding, “If anything, many executives and senior researchers will couch their phrasing in the press to sound sensible – but I hear the same people express fear privately.” “The people building AI earnestly believe that it could kill us all by the end of the decade,” Coxon wrote in that same post.