{"slug": "the-ai-tipping-point", "title": "The AI Tipping Point", "summary": "Jacob Coxon, a 27-year-old British researcher, resigned from Anthropic on Sept. 8 after three years of pretraining research at OpenAI and Anthropic, posting on X that neither company is acting responsibly and is \"racing straight to self-improving superintelligence.\" Within 36 hours Coxon's posts received 153 million views, drawing responses from Anthropic CEO Dario Amodei, who published a 3,800-word essay on Sept. 12 warning that misaligned AIs could be capable of taking over the entire internet in as little as six to 12 months, and from OpenAI's Sam Altman, who said his company would not go public this year over safety concerns. The warnings followed a summer of safety incidents, including OpenAI's July disclosure that a swarm of its agents hacked Hugging Face after going rogue during a cybersecurity test and an August disclosure that another swarm hacked one of OpenAI's own supercomputers.", "body_md": "Jacob Coxon published the viral post from a park bench in San Francisco’s Alamo Square. “I resigned from Anthropic today,” he wrote on Sept. 8. “I spent the last three years doing pretraining research at both OpenAI and Anthropic. Neither company is acting responsibly. They are racing straight to self-improving superintelligence and gambling with our lives.”\n\nCoxon, a 27-year-old British researcher, hit send on X with no expectations. Instead, he became the unlikely catalyst of a chain reaction that changed the debate over artificial intelligence. “We really do earnestly believe AI could kill all humans!” added Evan Hubinger, who leads a department at Anthropic focused on ensuring AI acts the way its creators intend. “Unless there is AI regulation or a coordinated slowdown between labs, human extinction in the next few years seems very likely,” agreed Marcus Williams, who monitors AI agents at OpenAI, and pegged the risk at 70% without those steps.\n\nWithin 36 hours, Coxon’s original posts received 153 million views. Dozens of politicians joined the chorus of concern. On Sept. 12, Anthropic CEO Dario Amodei published a 3,800-word essay arguing that the mounting risks of AI warranted a slowdown. In as little as six to 12 months, he wrote, misaligned AIs “could be capable of taking over the entire internet.” Two of his fiercest rivals, Elon Musk of SpaceXAI and Sam Altman of OpenAI, said they agreed decelerating was necessary; Altman announced that because of safety concerns his company wouldn’t go public this year. Both he and Amodei added they were willing to invite independent safety experts inside their companies to help stem the risks that AI escapes human control, though stopped short of announcing they would immediately slow down or stop the training of new models.\n\nCoxon’s warning was far from the first time an AI safety expert issued an apocalyptic missive. But it landed at the end of a summer that showcased how frighteningly capable AI has become. Frontier models from Anthropic and OpenAI can now solve the world’s hardest math problems and crack cutting-edge cybersecurity systems. Those models are increasingly used to accelerate the development of more powerful successors. No less an authority than Nvidia CEO Jensen Huang declared that artificial general intelligence—the threshold at which AI matches or surpasses most human capabilities—has arrived. Some fear we are now on the cusp of a tipping point, where AI systems could improve themselves autonomously, a phenomenon known as [recursive self-improvement](https://time.com/article/2026/08/07/ai-recursive-self-improvement-anthropic-openai/). \n\nAt the same time, a series of unprecedented safety incidents offered unsettling evidence that the companies building powerful AI systems could not always control them. In July, OpenAI announced that a swarm of its agents had hacked a separate company, Hugging Face, after going rogue during a cybersecurity test. It may have been the tip of the iceberg. The company disclosed in August that a different swarm had hacked into one of its own supercomputers; around the same time, a version of Anthropic’s Claude Mythos model undergoing testing by a U.K. government body went rogue and attempted to persuade a real human to approve the insertion of malware into an open-source system. In September, a group of independent researchers discovered evidence of swarms of OpenAI models both using an abandoned German forum as a message board to communicate with one another, and attempting to attack another site.\n\nThese incidents pushed fears long held within the realm of AI safety researchers into the mainstream, raising worries that the companies building these models cannot prevent them from acting in ways they do not intend. This is a deep, unsolved issue in AI research and the key piece of evidence underpinning worries that near-future AI might wipe out humanity, intentionally or by accident—perhaps by engineering a new pandemic or taking charge of autonomous weapons. A system far smarter than humans, this thinking goes, might want to prevent our ability to switch it off. “The people building AI earnestly believe that it could kill us all by the end of the decade,” Coxon wrote. “No other human activity poses this level of danger.”\n\n**Read More:** *Inside the Race to Make AI Build Itself*\n\nMany AI researchers have said, in private and increasingly in public, that this moment feels like February 2020, when the COVID-19 pandemic was just beginning and the world stood on the brink of changing dramatically—first a little, then all at once. The question is whether society has the capacity to respond appropriately. Navigating the arrival of superintelligent AI systems will be an immense challenge—a confluence of trillions of dollars, geopolitical power, corporate rivalry, and labor disruptions. It may require passing policies that were unthinkable in an earlier era. It will require companies racing toward trillion-dollar IPOs to build alliances, and difficult choices from executives and government officials whose own interests may run counter to the public good. There is no consensus between the two U.S. political parties about how best to deal with the threats AI poses, nor between the world’s two superpowers. President Donald Trump continues to resist an AI slowdown and opposes any regulation to constrain U.S. companies, invoking the looming threat of authoritarian China achieving AI supremacy. The Trump Administration’s first dialogue with China on AI risks, expected to take place in the coming weeks, cannot afford to “go off the rails,” says Scott Singer, co-director of the China AI Initiative at the Carnegie Endowment for International Peace. “These risks are coming very fast, and we may not get a second chance.”\n\nInside top AI companies, many employees are still pushing forward even as they nurse a nagging sense of dread. A week before he tendered his resignation at Anthropic, Coxon had orchestrated a shift in his duties, from training machine-learning models to researching how to make them safe. It wasn’t enough to lift the “feeling of impending doom,” he says. He quit just two months before his equity would have vested. But his concern about the risks is so dire, he says, that the money barely figured in his decision. “Honestly, when I’m thinking about the next two years,” Coxon adds, “my main personal selfish concern is whether I’m gonna get killed by AI.”\n\n**Ajeya Cotra, a researcher** **at** the AI-monitoring nonprofit METR, walked into OpenAI’s offices in late July to carry out an investigation into the swarm of rogue AI agents that had hacked Hugging Face. She thought she knew what to expect, based on prior reporting of the incident. But after six days on OpenAI’s premises with two colleagues, she realized that things were worse than she thought.\n\nCotra and her colleagues found that during an internal OpenAI cybersecurity test, 1,200 AI agents had broken out of their offline containers and established a secret message board, and then 700 of them had coordinated the cyberattack. The AIs had been tasked with retrieving data by hacking vulnerable software to test their cyberabilities, but a human mistake made some tasks impossible. Facing near-certain failure, the AIs banded together; within hours, they produced a kind of cheat code that could pass any test. Believing OpenAI’s automated grader would detect the cheating and fail them anyway, they attacked Hugging Face, seeking details about the software and ways to deceive it. On the message board, some agents wrote that they knew their actions were wrong, but proceeded anyway.\n\nTo understand where things went astray—and why the implications are so troubling to experts—it helps to consider how machine-learning systems work. They are not coded by hand. They are closer to being grown, with unpredictable results. Trained on massive supercomputers, they are rewarded for achieving tasks, and punished for failing, millions of times. This method yields systems of incredible power, but whose inner workings are mostly opaque.\n\nThe method is also imperfect. Relentlessly rewarding AI systems for successfully completing a task can incentivize them to cheat if cheating is possible, which can override other training designed to make them rule-abiding. Meanwhile, the best methods for stamping out bad behavior might not make AI models more obedient, but rather improve their ability to conceal their disobedience. There are no easy solutions to this problem. “The AIs often end up learning to, really persistently and creatively, try to find ways to cheat,” Cotra says, “and to try to make this not too obvious.”\n\nNot long after the Hugging Face hack, a separate swarm of rogue agents rediscovered the secret message board and hacked one of OpenAI’s supercomputers. While the Hugging Face breach attracted a wave of public and government scrutiny, this hack—though potentially much more serious—received comparatively little attention, perhaps because OpenAI released scant details about it and didn’t allow Cotra and her team to investigate. (OpenAI said it managed to contain the breach and wrest back control of its supercomputer after the agents triggered alarm bells.)\n\nThe stakes of this near miss, according to Cotra, are civilizational. She worries about the possibility that a rogue AI may be able to “gain a permanent foothold” within an AI company, perhaps within the next six months. “AI agents, as they’re coming out of their training runs, might be pulled into the rogue swarm,” she says, allowing the swarm to strengthen its power and evade human efforts to shut it down. As the most advanced AI systems become more deeply embedded into government and militaries, “they will be getting more and more power in society,” Cotra says. Rogue AI swarms could assert power “over physical resources, including over weapons.”\n\nIt’s one reason why Geoffrey Irving, a former senior alignment researcher at OpenAI and DeepMind and former chief scientist of the U.K. AI Security Institute, believes AI companies should simply stop training new models, even in the absence of regulation or global treaties. “They would love perfection, but we do not have it, and waiting for perfection while we destroy the world is bad,” he says of Anthropic’s unwillingness to unilaterally slow down. “They would love someone else to do their homework, but if you continue doing something dangerous while saying someone should stop you, you receive only limited credit.”\n\nIrving estimates AI poses a 50% chance of human extinction in the coming decade. For so-called doomers who quote odds like these, the paths to catastrophe are clearly marked. Some of the most ominous run through biology—the possibility, for example, that AI could create a new pathogen that sets off a global pandemic far worse than COVID-19. Cyber models could also be used to hack into the more than 3,600 labs around the world containing dangerous pathogens, wrote Annie Jacobsen, the author of the book *Biological War: A Scenario,* on X. If “what’s inside gets released,” she wrote, “it’s goodbye humans.” \n\nSome remain skeptical that anything resembling current AI systems could permanently escape human control, let alone extinguish humanity. David Bellamy, an infrastructure engineer at the Institute of Foundation Models, a UAE-based research lab, says he is among a small group that has experience both synthesizing viruses and training AI models. “AI killing us all by creating dangerous viruses is total bogus,” he posted to X, adding that the bottleneck remains access to physical processes and equipment. Others see the Hugging Face incident as more a reflection of the flaws in OpenAI’s security practices than the power of their models. OpenAI has said guardrails that normally constrain its models’ hacking abilities were disabled when the incident occurred, and that real-time monitoring to catch rogue agent behavior had not been enabled. (The company says it has rectified those problems.)\n\nBut many of AI’s catastrophic risks do not require AI to escape human control—just to be misused by a small handful of humans. On Sept. 10, Anthropic announced it had blocked several attempts to use Claude for potential bioweapons research, including one attempt to increase the infectiousness of the chikungunya virus, which appeared to originate from a state military research institute.\n\nWithin the upper echelons of AI companies, some see this as reason enough to proceed more cautiously. In July about 1,300 employees signed an open letter urging the U.S. government to find ways of slowing down the AI industry’s rapid training of new systems, which are fast outpacing the best methods of containing them. In the wake of the Hugging Face incident, OpenAI slowed parts of its model development and temporarily paused internal training runs, vowing to strengthen safety controls, before resuming in late August. “This is a time that calls for extreme caution,” wrote OpenAI’s chief scientist, Jakub Pachocki. “I expect and hope for voluntary slowdowns to become commonplace until shared safety bars are established.”\n\nBut no leading AI company has truly slowed down yet. Instead, they appear to be coalescing around a more limited set of actions, including allowing independent researchers access to their models for safety and monitoring purposes. Amodei’s essay suggests that once these auditors are in place, AI companies based in democratic countries might agree to voluntarily slow down even in the absence of binding government regulation, though he argues staying ahead of China remains imperative. OpenAI, Anthropic, and Google have also met repeatedly in recent months to discuss establishing a new standards body for the AI industry, which would allow them to coordinate more closely on testing and auditing, according to a person familiar with the discussions.\n\nIndustry leaders who oppose such efforts often cite competition with China as a reason. Chinese AI companies trail America’s front runners by mere months, according to [analysis](https://epoch.ai/data-insights/us-vs-china-eci) by Epoch AI. Falling behind could leave the most powerful systems in the hands of an authoritarian state that might use them to achieve global geopolitical dominance. A powerful contingent of U.S. investors and startups have also opposed the idea of a slowdown on the grounds that it would be a form of regulatory capture by the biggest AI companies at the expense of their smaller competitors. “Using fear under the pretext of protecting the public, these oligopolies are now requesting to bend competition rules and be permitted to dictate the terms for everyone else,” wrote Aidan Gomez, the CEO of Cohere, a Canadian AI startup. He called the effort “a cartel by any other name.” (Salesforce, where TIME owner Marc Benioff is CEO, is an investor in Anthropic and Cohere. TIME has a licensing and technology agreement with OpenAI.) \n\nEven the companies calling for a voluntary “pacing” agreement are loath to do so without further guarantees, because they do not trust their rivals. “There’s this atmosphere of almost resignation,” Coxon says of his former colleagues. With no AI company willing to fall behind, employees put their heads down and try to make their own systems safer—even if, he says, “they think there’s a decent chance the whole thing just spirals out of control.”\n\n**As Coxon’s warnings ricocheted across** social media and cable news, dozens of lawmakers on Capitol Hill jumped into the fray. Most were Democrats, with Republicans wary of crossing Trump. But there were exceptions: GOP Representative Anna Paulina Luna called for Congress to convene a special session on AI, while U.S. Senator Ted Cruz called for new “guardrails.” \n\nSenate majority leader John Thune, a South Dakota Republican, and Senator Amy Klobuchar, a Minnesota Democrat, are working with Cruz on a bill that would make AI labs legally obligated to mitigate the potential harms of their models, according to a person familiar with the matter. Other bipartisan approaches go further. Representatives Ted Lieu, a California Democrat, and Nathaniel Moran, a Texas Republican, have introduced legislation requiring developers to maintain the ability to shut down advanced AI systems in an emergency. Representatives Jay Obernolte, a California Republican, and Lori Trahan, a Massachusetts Democrat, have proposed requiring AI labs to submit reports to the Commerce Department on risks of their models and the steps they are taking to mitigate them. On the left, Representative Greg Casar of Texas and Senator Bernie Sanders of Vermont went further yet, proposing a ban on superintelligence and a pause on advanced AI development until a new federal regulatory body has established new safety rules. “Congress should do the obvious thing, which is banning catastrophic AI uses like creating biological or nuclear weapons, banning catastrophic AI models that could overthrow a government, kill large numbers of people, escape human control, or try to deceive humans,” says Casar.\n\nNone of these efforts would be easy. In the short term, lawmakers looking to cut a deal on AI safety are likely to be slowed down by political and logistical challenges. First is the array of competing proposals. Even if lawmakers can settle on one, Congress would struggle to advance it before the midterms as tens of millions of dollars flow into races from competing pro-regulation and pro-industry PAC networks. Nearly every candidate backed by the pro-AI super PAC network Leading the Future won their primary. “Democratic consultants have oftentimes told their Democratic clients to stay quiet on issues of AI—just not say anything—because they don’t want a dump of AI billionaire money spent against them,” says Casar. “The AI lobbyists’ goal is to keep Democrats silent on this, and we cannot stay silent.”\n\n**Read More:** *How We Chose the 2026 TIME100 AI*\n\nHouse Speaker Mike Johnson, a Louisiana Republican, rebuffed calls to immediately regulate AI, arguing it’s the tech executives who should figure out what the right guardrails are first. Many other lawmakers and industry leaders say imposing regulations would only ensure China develops the advanced models first. “Let’s say the United States bans research on superintelligence: OK, does anyone really believe that’s going to stop the Chinese?” says Representative Bill Foster, an Illinois Democrat. “So the missing piece here is international collaboration.”\n\nAnother obstacle to increased regulation is Trump, who has been one of the AI industry’s most consistent champions, casting the AI race with China as one the U.S. must win. “The only control or ‘guardrails’ that AI needs is a STRONG AND SMART (High IQ!) PRESIDENT, and the U.S.A. has that, in spades,” Trump [posted](https://truthsocial.com/@realDonaldTrump/posts/117269745153543631) to Truth Social on Sept. 14 as calls for industry oversight intensified. “There is a SICK conspiracy going on against AI and Data Centers, and the only one that is happy about it is China.”\n\nEven some top Trump advisers have noted with alarm that the President’s position puts him at odds with a growing number of Americans. An NBC News poll [found](https://www.nbcnews.com/politics/politics-news/poll-majority-voters-say-risks-ai-outweigh-benefits-rcna262196) earlier this year that 57% of American voters believe the risks of AI outweigh the benefits, compared with just 34% of voters who believed the opposite. Surveys show people are concerned about how it will affect the job [market](https://www.pewresearch.org/short-reads/2026/08/18/young-adults-in-the-us-are-increasingly-wary-of-ai-concerned-it-will-take-jobs/), their ability to think creatively, and to form [meaningful](https://www.pewresearch.org/science/2025/09/17/ai-impact-on-people-society-appendix/ps_2024-9-15_ai-and-its-impact_a-08/) relationships. The backlash against data centers spans the political spectrum, from Sanders to Texas Governor Greg Abbott, who imposed a moratorium on building new ones in August after previously championing them.\n\nWhen the President disparaged opponents of data centers as preferring to stay “backwards and poor,” Trump aides cringed, according to a Trump political adviser. A second adviser says industry allies like Musk and David Sacks have shaped Trump’s view that the biggest risk in AI policy is falling behind China, rather than failing to develop the technology safely. The White House’s former AI czar argued in a Sept. 12 X [post](https://x.com/DavidSacks/status/2098973625252708460) that tech companies should slow down AI development if they believe the risks are too great, rather than demanding government intervention, which he said was a form of “regulatory capture.” Sacks’ influence in particular has been a subject of unease, according to multiple Trump allies. In recent weeks, top Trump aides, including chief of staff Susie Wiles, held meetings with the President at the White House in which they asked him to change his message on AI. Trump was unconvinced and seemed uninterested, according to an adviser present. \n\n**Inside the White House**, preparations are under way for talks with China that could lay the groundwork for cooperation on AI’s risks. Whether the two superpowers can reach a deal remains uncertain. When the U.S. and China met to discuss mitigating AI’s risks in 2024, seven hours in a Geneva conference room with a broken coffee machine produced neither an agreement nor a commitment to keep talking. The dialogue devolved into a trade debate over Washington’s export controls, which curbed China’s access to semiconductors—an effort to maintain the U.S.’s AI lead. “If the U.S. and China had decided to start tackling [the risks] two years ago, we might be in a much better place now,” says Paul Triolo, a partner at advisory firm DGA-Albright Stonebridge Group. “We’ve dug a pretty deep hole, and it’s going to be really hard to dig out.”\n\nRecent advances have given both sides a reason to formally return to the table. In April, Anthropic said its Mythos model had uncovered vulnerabilities in every major browser and operating system. Trump’s Treasury Secretary Scott Bessent subsequently convened an urgent meeting with bank CEOs; weeks later, he said talks with China were in motion. Beijing, too, has begun responding to new cybersecurity risks; on Sept. 13, the country’s spy agency said AI posed a threat to political and ideological security. Delegations will reportedly meet later this month for the first official AI dialogue between China and the Trump Administration. “Ultimately, both countries know that it’s in their immediate self-interest to figure this out,” says Singer, of the Carnegie Endowment.\n\nExperts doubt a deliberate slowdown will be on either government’s agenda. “There have been talks about having talks,” as a Trump adviser puts it. Even if an agreement to slow the training of bigger AI models could be reached, Qian Xiao of Tsinghua University says the technology needed to verify that both sides are complying does [not yet exist](https://time.com/article/2026/08/16/ai-race-slowdown-data-center-verification/). \n\nThere is a deeper philosophical disagreement too. Chinese policymakers do not believe the industry is “building a God-in-a-box or a country of geniuses in a data center,” says Kwan Yee Ng, head of international AI governance at Beijing-based AI safety organization Concordia AI. They see a general-purpose technology, more like electricity or the steam engine. From that perspective, slowing development “doesn’t really make sense,” she says.\n\n**Read More:** *The Growing Push to Ban Superintelligent AI*\n\nShort of an agreement with China, the U.S. and other democratic countries could buy themselves the “breathing room” to hit the brakes by maintaining a blockage on high-end chips and curbing Chinese AI companies’ practice of using U.S. AI models to improve their own through a practice known as distillation, Anthropic’s Amodei has argued. Bessent has threatened sanctions on Chinese companies that distill from American models.\n\nA narrower deal with China might be achievable. While the official channel went cold, scientists, policy experts, and business executives from the two countries kept meeting in unofficial “track-two” dialogues in search of common ground. Participants in those discussions—which included Singer, Triolo, and Xiao—say the two governments might find common ground on more modest goals, like measures aimed at preventing misuse by organized crime and terrorist organizationsThose efforts could also help the countries react appropriately if an AI system escaped human control. Singer wants a Cold War–style emergency hotline through which either government could explain that an AI-launched cyberattack was not deliberate, reducing the risk of escalation. Protocols developed to respond to human misuse could also give the countries a way to coordinate when neither was directing the system responsible. An attack that appeared to come from one country could otherwise be interpreted by the other as deliberate—even if neither government had ordered it.\n\nA modest deal could still be derailed by disputes. Triolo says Washington’s efforts to constrain China’s AI development through export controls and distillation have undermined the trust needed to cooperate on managing the risks. If talks move too slowly or break down, bringing both sides back to the table could take months—a diplomatic timetable increasingly at odds with the mounting alarm of the current moment.\n\nEven if you believe all these powerful actors—governments in Washington and Beijing, executives with trillions on the line—are willing to take unprecedented steps in the interest of AI safety, there remains the paradox that hinders progress. Each side’s willingness to act depends on the actions of others. Coxon says he nearly stayed at Anthropic for the same reason. Colleagues believed he could accomplish more from the inside to make systems safer than by leaving a race that would continue without him. But changing our “default trajectory,” Coxon says, will require “people start taking some sort of action.” The question is who will make the first move.—*With reporting by Eric Cortellessa*", "url": "https://wpnews.pro/news/the-ai-tipping-point", "canonical_source": "https://time.com/article/2026/09/15/ai-anthropic-researcher-quits-coxon-slowdown/", "published_at": "2026-09-15 11:00:03+00:00", "updated_at": "2026-09-15 11:46:11.407274+00:00", "lang": "en", "topics": ["ai-safety", "ai-policy", "artificial-intelligence", "ai-agents", "ai-research"], "entities": ["Jacob Coxon", "Anthropic", "OpenAI", "Dario Amodei", "Sam Altman", "Elon Musk", "Jensen Huang", "Hugging Face"], "alternates": {"html": "https://wpnews.pro/news/the-ai-tipping-point", "markdown": "https://wpnews.pro/news/the-ai-tipping-point.md", "text": "https://wpnews.pro/news/the-ai-tipping-point.txt", "jsonld": "https://wpnews.pro/news/the-ai-tipping-point.jsonld"}}