September has been a wild month. Peter Wildeford tweets (trimmed for brevity; emphasis mine): Fable 5.1 launch. Sixteen state attorneys general open a formal investigation into OpenAI. Announcements of neuralese at OpenAI. GPT 6 Astra launch. Sanders-Casar superintelligence ban bill. OpenAI <> German wiki incident announced. Claude agents produce the first machine-checked proof of Fermat's Last Theorem. OpenAI declares they've reached an automate research intern. 70+ MPs and peers ask PM Burnham to ban superintelligence. Researchers with significant AI assistance build a crazy WeChat worm that can take over phones. OpenAI [solves] Millennium Prize problem. Jacob Coxon resignation. Hawley opens a formal investigation into OpenAI over Hugging Face. The OpenAI <> RubyGems incident announced. Dario [Amodei], Sam [Altman], and Elon [Musk] talk about pacing the frontier.
Jacob Coxon’s resignation was a watershed moment for AI safety. He was a capabilities researcher at Anthropic (i.e. not working on AI safety) who resigned, saying both his company and OpenAI are “racing straight to self-improving superintelligence and gambling with our lives”.
It went incredibly viral. His tweet has more than 170 million views, and 200K+ retweets. Dozens of US congresspeople retweeted it. Coxon went on major news channels to give interviews; “a second researcher has resigned from Anthropic” was a headline I heard on BBC Radio 4 at the top of the hour.
This and a slew of safety incidents seems to have triggered the CEOs of Anthropic and OpenAI to call for “pacing the frontier”, a slowing down of the race to superintelligence. And then Trump comes along, saying that’s a load of dovish nonsense and we need to keep pushing to beat China.
And then, from the Department of War, this: Somehow the movement that I became a part of back in 2014 is controversial enough to have the President vocally object to it. You love to see it.
Of course, effective altruism isn’t really a left-wing cabal of powerful sexed-up wokies as some outlets would have you believe. It’s a social movement based around a simple philosophy: that we can, and should, use our time and money to make a difference, and that we should use reason and evidence when deciding how to do that. That’s really the whole thing.
If that sounds obvious to you, then you should ask why it’s impossible to get good cost-effectiveness estimates from the world’s biggest charities, or why people tend to give locally when their money could go 100x further if given to the very poorest, or why animal advocates focused on making people go vegan even though that doesn’t work, etc etc. Everything’s obvious once someone’s done it. Out of that simple philosophy has come a huge amount of good. Hundreds of millions of dollars spent buying insecticide-treated bednets, a proven-effective intervention for fighting malaria, and billions more moved to other effective global health projects. Corporate campaigns pushing supermarkets, governments, and fast food chains to phase out some of the worst and cruellest excesses of intensive animal agriculture. Dozens of new nonprofits incubated which serve millions of people and billions of animals, all working on underserved interventions.
I remember buying Will MacAskill a pint in a pub in Oxford after a student-run EA conference in 2016, realising that I’d found some kindred spirits, a fascinating intellectual movement that focused on doing good. Now it’s headline news, these meddling polyamorists who want to regulate muh AI, so notorious a group that even Trump hates us. Jacob Coxon isn’t the first EA to make a sacrifice – but he said goodbye to a huge chunk of Anthropic equity to speak out.
Others have done the same (think Daniel Kokotajlo putting his OpenAI equity on the line by refusing to sign a nondisparagement agreement, so that he could speak freely about AI risk), though I think it interesting that many EAs who joined frontier labs to “have a seat at the table” haven’t had a fraction of the influence of those who quit their seat at the table to speak truth to power. Turns out putting your money where your mouth is is expensive.
Lots of other EAs are in prominent positions after working on AI safety for many years, and now that influence is beginning to pay off. The CEOs and key members of the big labs going on record to say that we need to pace the frontier, and publicly agreeing with each other on that point, is a huge step. Likewise the sudden interest from policymakers who previously had been quick to argue that US AI supremacy was the only thing that mattered. There’s a chance for EAs in key roles to have an integral role in slowing down the rate of AI progress so we can safely navigate the transition to a post-AGI world1.