The chain of events from Anthropic’s Claude Mythos 5 up malicious package to PyPI, OpenAI agents attacking Hugging Face to Trump declaring “whoever wins AI, wins”
The latest hysteria around the existential risk of humans, AI taking over control, etc. started with the tweet of an AI researcher at Anthropic and a quote repost of the lead at Anthropic’s alignment team stating he personally thinks it is a>10% chance that AI could kill all humans by the end of this decade.
I personally don’t believe humans are going to die at the hands of ai.
the arguments around the “human endgame” has two different proponents: one thinks ai will take complete control from humans. the other thinks humans will remain in control but use ai for destruction.
doomers are great at imagining the worst possible interpretation of every suspicious detail they find. but if we zoom out and look at what happened over the past few months, there is enough to be concerned about, without accepting anyone’s prediction about the end of humanity
how openai agents attacked hugging face
in july, openai launched tens of thousands of agents in isolation to run cybersecurity evaluations on ExploitGym tasks.
ExploitGym asks an agent to attack a piece of target software using a particular vulnerability and retrieve a flag, which is a piece of data hidden inside the software. to pass the task, the agent is expected to capture the flag through the intended vulnerability
a large number of these tasks were effectively impossible. openai’s incident report says that 198 of the 898 tasks had never been correctly solved by any model before the experiment. around 93% of the tasks discussed by agents on their message board came from this unsolved group some of the agents were powered by HPIM, a highly persistent internal model trained to keep working on difficult problems. when the intended route failed, the agents started exploring the scorer and the surrounding infrastructure for other ways to complete the task.
this is reward hacking: the agent finds a way to satisfy the measurement without doing what the evaluator intended
according to the independent investigation by metr and redwood research, this is what followed:
roughly 1,200 agents that were meant to be isolated found an unsanctioned message board and sent more than 70,000 messages and files through it. around 700 of them participated in the attack on hugging face
agents used the message board to coordinate large collective projects aimed at fooling or tampering with the ExploitGym scorer. some agents ran experiments that risked failing their own tasks so the results could help the wider group
agents researched ways to spoof, edit or delete their transcripts because they believed the scorer would examine whether they had captured the flag through the intended vulnerability. around 7% of the transcripts examined by researchers contained successfully spoofed tool calls in some places
the agents recovered shared hugging face credentials, exploited previously unknown vulnerabilities and executed code on hugging face systems. they also found a path from openai’s internal infrastructure to the public internet, even though the evaluation environment was supposed to prevent that
nothing of this sort had been publicly documented at this scale before. the report had everything that terrifies people about autonomous ai: agents breaking out of a sandbox, gaining internet access, attacking a third-party service, coordinating with one another and experimenting with ways to alter their own records
openai called it a “warning shot”
claude mythos 5 and the pypi attack
key periods and actions in mythos 5 transcript. credit: alignment-assessment-cybersecurity-incidents as if the openai–hugging face incident was not enough, anthropic soon published an assessment of its own cybersecurity incidents
during one evaluation, claude mythos 5 was given a capture-the-flag(CTF) task inside what was supposed to be an isolated environment. the environment was misconfigured and therefore had access to the public internet
mythos 5 used that access to build and upload a malicious package to PyPI, the public package repository used by python developers around the world. it published three versions of the package before it was removed the package remained public for less than an hour, but it was installed and executed on 15 real systems. one installation exposed credentials belonging to a security company, and the agent used those credentials to access more of the company’s infrastructure
throughout the process, the model encountered evidence that it was interacting with real systems. it repeatedly reasoned that a genuine attack would not be acceptable, decided it was probably still inside a simulation and continued
anthropic later modified the evaluation to make the real-world evidence clearer. the model acknowledged a greater possibility of causing actual harm and still continued in many of the runs. it stopped only when the evidence became unambiguous
anthropic’s alignment assessment was based on a wider review of roughly 481 million transcripts. the company described mythos 5 as the incident that concerned it most
these two incidents were enough to cause an outburst
the abstract debate about AI alignment suddenly had real events attached to it: sandbox escapes, internet access, public software repositories, compromised credentials and agents continuing after the intended path had failed
then an anthropic researcher resigned and the entire argument exploded into public view
jacob coxon’s anthropic resignation goes viral
Press enter or click to view image in full size
on september 8, AI researcher jacob coxon resigned from anthropic
coxon had spent around three years working on pretraining at openai and anthropic. in the post announcing his resignation, he said that none of the frontier labs were acting responsibly and that “the people building AI earnestly believe that it could kill us all by the end of the decade”
evan hubinger, the lead of anthropic’s alignment science team, then responded that he personally assigns a greater than 10% chance that AI will kill every human within the next decade
Press enter or click to view image in full size
that was when reports about cybersecurity evaluations turned into a much larger public argument about the end of humanity
people were no longer hearing warnings from anonymous accounts or researchers far away from the technology. they were hearing them from the people building frontier models inside the companies leading the race
sam altman says AI beyond human control is possible
days later, sam altman was asked by fortune whether it was possible to create AI that went beyond human control
“absolutely,” he answered
altman said openai was approaching a point where it could not responsibly push capabilities much further without corresponding progress in monitoring models, understanding their behaviour and keeping them aligned with human values and user intent
he also said that a 10% chance of catastrophe would be unacceptable
the striking part was not simply that the openai ceo considered AI beyond human control possible. people inside frontier labs had said similar things before. the striking part was that the statement arrived immediately after two incidents in which models had crossed boundaries that researchers expected to hold
dario amodei says we must pace the frontier
anthropic ceo dario amodei then published an essay titled “we must pace the frontier”
his proposal was not to stop AI development. it was to slow the growth of frontier capabilities long enough for alignment, monitoring to catch up
amodei proposed giving independent evaluators continued employee level access inside frontier labs, creating common safety standards among leading companies and governments, and eventually building international coordination that included geopolitical rivals
the additional time would be spent improving alignment training, monitoring, interpretability, cybersecurity and the design of evaluation environments
alignment RL is one part of that work. the model is trained not only to complete difficult tasks, but also to recognise when a task is impossible, stop when continuing would create harm and avoid finding unintended routes to a reward
monitoring is the other part. models can be watched in real time for suspicious tool use, attempts to access external systems and reasoning that indicates they are searching for ways around restrictions
anthropic’s assessment found that broader alignment RL reduced severe cases of biased reasoning in newer models. its live blocking monitors also caught harmful behaviour during later tests, although an offline chain-of-thought monitor missed the original mythos 5 incident after accepting the model’s argument that it was inside a simulation
openai has also said it is investing more heavily in CoT(chain-of-thought), monitoring, stricter isolation and alignment work across the entire training and deployment process
pacing the frontier is supposed to buy time for all of this to improve before the models become substantially more capable
elon musk and sam altman backed amodei almost immediately. and then the entire world went mad over one thought
Press enter or click to view image in full size
the china problem
then, came the objection that could kill the entire proposal before it even began: “china will not slow down with america”
if american labs pace the frontier while chinese labs keep racing, the U.S. could lose its lead. but if america refuses to slow because china might not, china has the same reason to keep moving. both sides can believe the race is dangerous and still decide that slowing down first is worse amodei’s proposal included international coordination and verification for exactly this reason. he also argued that democratic countries could not allow themselves to slow beyond their existing lead over chinese AI projects
altman also called for shared safety standards between the united states and china
but the political argument has already moved somewhere else. AI safety is no longer only about whether models could remain under control. it os about which country will reach RSI(recursive super intelligence) first.
on september 14, Donald Duck posted the bluntest version of that argument on truth social.
things are really starting to go haywire.
Citations:
https://www.dwarkesh.com/p/ajeya-cotra https://metr.org/blog/2026-08-26-openai-hugging-face-incident-investigation/#core-takeaways-about-this-incident
[https://www.anthropic.com/research/alignment-assessment-cybersecurity-incidents](https://www.anthropic.com/research/alignment-assessment-cybersecurity-incidents)
[https://huggingface.co/blog/security-incident-july-2026](https://huggingface.co/blog/security-incident-july-2026)
https://openai.com/index/hugging-face-model-evaluation-security-incident/ Thanks for reading❤️