This has now been informed by the events surrounding OpenAI training models for months while they had access to a joint de facto message board, which was detected only in the wake of the hacking of HuggingFace by OpenAI’s AIs models during a cybersecurity eval. As we find out more about that, a lot of people have grown far more alarmed, as they should given what they previously believed about the difficulty of alignment, about the state of capabilities and about the level of operational supervision, infrastructure, safety and safety culture at the frontier labs.
This post will not go further into the details of that incident. It treats that as background to keep in mind, and mostly involves perspectives from before the Black Hat talk. This was originally scheduled for Friday and got bumped.
A lot of the disagreements about the need to pace tie into expectations about the default pace of capability advancements. As I wrote recently in The Three AI Pills, sincere disagreements about AI policy usually boil down to disagreements about the expected pace of progress, and what we expect future AIs will be able to do.
There are also disagreements about how dangerous a given level of capability would be, or how it would physically impact the world, and disagreements about what options we have, the nature of various coordination mechanisms or government interventions, and balancing different sacred values. Often people only properly see one half of a key trade-off.
Mostly the reason people often sincerely only see one half of such key tradeoffs is that they anticipate so little AI progress that mitigating existential risks or worrying about humans losing control is unnecessary.
Or they anticipate so much AI progress that mitigating existential and catastrophic risks and maintaining human control has to be the priority, and that necessarily is going to mean some group of people collectively choosing and charting, in some way, a deliberate path through causal space towards outcomes that allow us to survive.
Yes, that necessarily means enabling some group of people to have some collective mechanism to chart some aspects of Earth’s path through causal space, and yes there are reasons to worry about that, but that is why we should work to figure out the best way to do that.
Progress Fast and Slow
Those opposing the Pacing the Frontier letter do not want to ‘slow down’ and instead want AI to go ‘fast,’ but their vision of fast is, while super fast by historic standards, not all that fast.
Those who signed the Pacing the Frontier letter mostly want AI to go at least as fast as the opposition. What they want to avoid is AI going super duper ultra hyper fast.
Daniel Eth (AI Safety): Here’s how I’m interpreting the words: : step on the brakes
Pace: don’t attach a giant fucking rocket engine on the back of the car that will accelerate us from 65mph to 10,000 mph, at least not unless we can turn it off. also, don’t disable the brakes.
Almost everyone signing the letter wants better chatbots and doctors and science and other forms of diffusion. They don’t want superintelligence and a singularity in 2027.
Nick: a lot of the anti crowd wants better chatbots and doctors and stuff, a lot of the pro crowd expects like way crazier worlds in the short term and wants them to come in the just slightly less short term when we’ve figured out how to control these things better. plenty of overlap
how to do it no idea, and also how to measure the speed limit no idea, but i feel like this framing avoids some of the issues with , which also has roughly the same questions
Dean W. Ball: This is what most people I know who signed the “pacing” letter (myself included) think. The slowdown we have in mind is temporary, and to a rate of progress that is still much faster than even today’s rate.
Augmented Fifth: It’s Dec 19, school is out, and the Christmas presents are under the tree. “Let’s wait until February” is not going to fly with the kids.
I quote that last one because what is happening is that the people who are arguing against the Pacing the Frontier letter are saying they won’t wait until February and demanding they get the presents on December 25 and they’d better get that BB gun.
Whereas those signing the letter are saying maybe we should wait until at least tomorrow before we order even more presents and there is no more room left in the house, plus maybe keep the toys reasonable, and only get you the BB gun that’ll shoot your eye out kid and not an AK-47 or tactical nuke or that sexy Von Neumann probe.
Drake Thomas (Anthropic): Very happy to have signed the pacing the frontier letter; its existence gives me a lot of hope for humanity’s survival!
I endorse the letter as written (and think, given the constraints, it’s probably close to the best it could be for this level of consensus), but some places where I differ from its connotational tone:
(1) Not only is AI “not guaranteed” to make a dramatically better future, the odds of failure are terrifyingly high: I think* there’s something like a 40% chance we get an outcome around as bad as human extinction or worse, and another 30% chance we get a future that, while containing some good things, falls radically short of what a wiser civilization could have obtained (say, <5% of the value of a truly great future).
(2) I don’t really endorse the vibes of “to realize AI’s potential”; I think a much more immediate and pressing motivation is “to avoid catastrophically bad outcomes from AI that will kill a lot of people”. (The potential is also very important, ofc, but I think most moral theories would view it as being of secondary importance when risks are this high and there’s little that would do more than temporarily delay that potential anyway.)
(3) “address emerging risks, develop security measures, and strengthen oversight” is fine so far as it goes but a little vague. Concrete things I’d like to do with slower AI development: way better interpretability, build up a robust and well-resourced third party ecosystem for independent auditing of AI companies and get lots of reps in for their oversight, put tons of effort into the automation of alignment research, work on governance mechanisms for ASI, develop a vastly better science of the nature and development of AI behavior, build much more powerful control mechanisms, point lots of powerful AI labor at ambitious scalable alignment projects (eg work like ARC’s), deep dives (including external audits) of individual AI behavior incidents, etc. Also getting civilizational biosecurity preparedness in order.
*epistemic status very approximate vibes, my numbers will change day to day and depending on the exact operationalization.
And a very good statement about the actual thing we may be on the verge of doing, and need to prepare to handle, from Samuel Hammond. You get results like this if you keep drawing straight lines on logarithmic graphs and follow the logic of everything involved. You can invoke the absurdity heuristic or ‘bottlenecks’ or what not all you want, but this is what the labs actually expect. The variation Hammond offers has things accelerating quite a bit but is not even fully ASI (superintelligence) pilled about the ultimate ends of this cycle:
Samuel Hammond: On the contrary, I’d argue liberalism originated in the Hobbesian moment when we jointly deferred to a higher power to preserve our agency and avoid killing each other.
Before succumbing to the temptation to naval gaze into the political theory abyss, it’s worth stepping back and clarifying what exactly is happening and being proposed.
Several US companies are on the precipice of fully automating the AI R&D loop, inclusive of pre/post training, env creation, data generation, evals, algorithm and kernel design, systems engineering, architecture search, etc. — the full stack.
We are already in a regime of weak RSI via partially automated SWEs, but closing the loop altogether represents a difference in degree becoming a difference in kind. The pace of progress will be explosive and potentially uncontrollable.
The US companies closest to this threshold are warning that they are unprepared for a runaway intelligence explosion, and yet feel locked into a prisoners dilemma vis a vis each other and to a lesser extent vis a vis China.
We’ve already seen how rapid and comparatively unbounded progress is in verifiable RL domains, leading to spikey forms of superintelligence in math and cyber, including models that can prove open math conjectures, discover massive speed-ups for breaking encryption, and execute sophisticated multi-step exploits. We’ve also recently seen several severe examples of “loss of control” / misalignment incidents given inadequate monitoring and sandboxing practices relative to model capability. Moreover, these new capabilities mostly stem from scaling-up long-horizon post-training on legacy clusters, with OOMs of new compute about come online / in construction.
In the pre-RSI regime, human frictions created automatic buffers between new model releases, giving researchers and society time to probe emergent capabilities, design better evals, develop novel alignment techniques, and adapt / harden their infrastructure. As progress has accelerated, capability improvements have already started outstripping our adaptive capacity, as manifest in METR’s inability to evaluate model autonomy beyond 13 hours, and narrow window for cyber defenders to prepare for open weight versions of Mythos.
RSI will exacerbate all these issues and create all new ones. At minimum, we should anticipate – the equivalent of a GPT-5.2 -> 5.6 leap in capabilities at least every 24 hours (down from 3-6 months), – concurrent algorithmic improvements densifying models to ultra-efficient sizes at any given capability level – 100x Mythos-like capabilities across most verifiable domains, including chem, nuclear and bio – new forms of multi-agent misalignment risk – “company in a box” agents trained to stand-up whole organizations / corporations – “cyber nuke”-like capabilities that require de minimis infra – several transformer-scale breakthroughs, such as for long-term memory / continual learning, open-ended domains, and/or all-new training techniques for idealized “GPT-zero”-esque metalearners – concurrent speedups in any complementary technical domain, i.e. explosive rates of R&D and novel discoveries
It seems to me there is little to lose, and much to gain, from having the social technology to “pace” these developments rather than to let them rip with zero industry / gov’t coordination, particularly as there is technically no law explicitly prohibiting a company from letting an RSI loop run indefinitely and unleashing whatever comes out the other end into the world.
There are innumerable ways an uncoordinated intelligence explosion could become an unmitigated disaster for the cause of liberalism, including runaway power concentration, rapid societal destabilization, rogue AIs / loss of control scenarios, WMD mass proliferation, vulnerable world technologies, and beyond.
Human civilization is about to be forever changed regardless, however if were possible to coordinate the handful of key actors and create artificial “buffers” between each step-change in model capability to enable adaptation, mitigation and alignment research to catch-up, it’s worth a shot.
Given the short-timeline, I think a DPA 708-style agreement is probably our best bet, i.e. an industry consortia with narrow antitrust carveouts for sharing safety and security practices, funding an assurance nonprofit / independent verification organization for 3rd party evals, incident reporting, internal deployment monitoring, standards setting, and enforcing a protocol for coordinated delays / slowdowns, among other things. This still leaves open the China question but that’s a bridge we won’t cross until after solving the collective action problem at home.
I’m open to other approaches / coordination frameworks but this is the object level issue we’re facing. Political theory is great, and I would love to use our limited steering capacity to guide AI development toward a future that maximizes individual liberty, but as a discussion baseline, gesturing at philosophical abstractions is simply non-responsive to the crisis at hand. A red-herring at best, a suicidal circlejerk at worst.
Samuel Hammond: If you do a simple linear regression on model release cadence it predicts a new frontier model will be produced roughly every day by January 2027. Whether a daily release cadence makes any sense is another question. By that point I suspect new models will be private by default.
The most important point here is up top. If things accelerate as described above, I think humanity is probably toast. Even if humanity survives that scenario, your liberal order is most definitely toast. It will seem absurd or incoherent to suggest otherwise, in a world with AIs that are more capable than humans across basically everything and growing more so every day, that have not been meaningfully constrained and cannot be collectively steered.
No One In Charge
Do you think that by ensuring no person is in charge of what the labs do, when the top lab is many current cycles ahead of the next one and the AIs outperform the humans across the board, that you will get a nice, friendly, liberal order of humans?
My actual read is that the thinking is ‘I don’t need to think about that, I just know that what you are proposing sounds bad, so I am against it.’
That reaction is a rejection of the premise of the question. It is refusing to be ASI pilled, even within a hypothetical.
In which case, one should state they are rejecting the premise, but also still answer the hypothetical. If such AIs did come to pass relatively soon, what do you think would happen by default? If you think the answer is still ‘nice, friendly, liberal order of humans,’ then what limitations are you still counting on to ensure this? What would cause it to break down?
There is extensive analysis of these options at the link. I see the logic in these proposals. I am most interested in option 4, to require safety cases, even though it is harder to implement. Option 2 also appeals (with ideal numbers TBD), to have a minimum compute allocation for alignment and safety efforts, and can be a complement, with the caveat of obvious problems pinning down what that means.
Pausing the Frontier
One can also take the full position, not taken by the Pacing the Frontier letter, that given recent events the frontier capabilities development should be d now.
This is importantly not what the Pacing the Frontier letter calls for. The Pacing the Frontier letter calls for gaining the capability to pace development later. Those calling for a want to set the pace of development to zero, right now.
As always there are three basic objections to this:
Coordination is too hard, incentives do not work, we cannot do it.
Think of the potential, we cannot afford to do it.
There is not enough potential, we do not need to do it.
Plausibly we are now entering the phase where it becomes so dangerous to continue that it is better to individually, even without coordination. Which would then make it far easier to coordinate.
Geoffrey Irving: I don’t think it is rational for anyone to be doing capabilities research at a frontier lab right now. We are not in a Prisoners Dilemma: the situation is very dangerous, and if one person or lab stops it makes it easier and more peer-compatible for other people or labs to stop.
A useful clarification is that this conclusion that unilateral stopping is rational is not obvious nor some kind of tautology: it depends on how dangerous the situation is. We’ve seen a lot of danger recently!
Industry coordination is way better than that! U.S. federal laws could be better still! International treaties are even better than that! But those might take time, even years, and it is not clear we have that time.
At some point, ‘you cannot afford to continue’ overwhelms ‘you cannot afford to stop,’ even if both are importantly true. It is now a lot more arguable that we are there.
I do not think we are at that point yet where I agree with Irving, but I am a lot less confident about this than I was two weeks ago. I would not be okay working on resuming capabilities development at OpenAI without assurances well beyond what we have seen in public, and am very glad they are consciously slowing development down.
This is distinct from not releasing things already developed, including GPT-5.6-Cyber and Project Daybreak, which is fine so long as GPT-5.6-Cyber at no point was trained with the message board active.
Almost every day, there is a new story about how your companies are losing control of the AI technology you are developing, with potentially cataclysmic results.
This week we learned, frighteningly, that AI has been used for the first time ever to create new viruses. As you know this type of development, in the wrong hands, could lead to new bioweapons that result in the deaths of tens of millions of people.
Last month, the world found out OpenAI lost control of an AI model. The result? The model hacked into another company’s computers—a clear violation of federal law. After conducting internal reviews, Anthropic and Meta reported their models similarly escaped their control.
One of the targeted companies called the AI hack “an unprecedented event” that deserves an “unprecedented response.” Yoshua Bengio, the most cited living scientist in the world, said these incidents “should serve as a wake-up call.”
I agree.
So do the top scientists at the companies you lead, the very people building this technology. As you know, these technology leaders recently called for the international community to create a safety mechanism—a button—to avoid catastrophe. They warned there is a “real risk that capability development rapidly accelerates beyond our ability to understand or control the resulting systems.”
And yet, at a moment when we have seen human loss of control and the creation of potentially dangerous viruses, your companies are still racing ahead, investing tens of billions of dollars into a technology that nobody can fully understand, predict or control.
That is absurd, irresponsible and extremely dangerous. It is also a betrayal of your own stated commitments:
In 2023, Anthropic said it would “commit to the scaling and/or delay the deployment of new models whenever our scaling ability outstrips our ability to comply with . . . safety procedures.”
In 2025, Meta said “if a frontier AI is assessed to have reached the critical risk threshold and cannot be mitigated, we will stop development.”
That same year, OpenAI said it would “halt further development” until strong safeguards were in place if AI capabilities ever reached a “critical” threshold.
That moment is here. AI capabilities HAVE reached a critical threshold. There is a reason why the head of the CIA says that AI models are “akin to digital nuclear weapons” and “almost like a doomsday device.”
Mr. Altman, Mr. Amodei and Mr. Zuckerberg: In the interest of humanity, stand by your words. AI development. It is not too late to avoid disaster. Stop building machines that humans cannot control.
Let me be very clear: If you do not take appropriate action now, my colleagues and I in the U.S. Senate will.
Sincerely,
BERNARD SANDERS
UNITED STATES SENATOR
OpenAI actually invoked the critical threshold days ago with Astra. They locked down that model and are taking some enhanced precautions. Good. What they are not interested in is an extended , and Altman made clear he still intends to push to release Astra soon.
Moderate Prudence
Dean Ball clarifies his view, that ‘with even moderate prudence, things will probably go extraordinarily well,’ but it is not so easy to have moderate prudence. Moderate prudence means entire organizations and tons of smart people coordinating to make things go well, and some believe humanity cannot muster even such modest efforts.
This is because Dean Ball is AGI pilled but he is not ASI pilled. He does not expect superintelligence to take the form I and others expect it to take.
If he is right about that, then I would not be as optimistic as he is but yes modest prudence would put us in a strong position. We would first need to overcome all the barriers to achieving modest prudence, as we are not currently on track to that. Instead, we are looking at high level fire alarms and staggering incompetence across the board, without much coordination.
Dean Ball’s statement below is a way of saying, correctly, that many people are not AGI pilled, and that this causes them to dismiss AI as ‘nothing new’ and otherwise be unwilling to risk the cost of even ‘moderate prudence.’
Whereas those who are AGI pilled understand we need at least moderate prudence.
Preparing to Pace the Frontier is an example of one aspect of moderate prudence.
Dean Ball: a collective effort to exercise “moderate prudence,” in a global civilization as capable as ours, is equal to more mindpower and strength than all of humanity combined up until very recently. “moderate prudence,” exercised across the whole of advanced civilization today, is a mightier thing than many ancient armies.
some people believe humanity cannot muster even a modest effort, but that our failure won’t matter, because ai is nothing new. it was never necessary to worry about a transformation because no transformation will occur, they believe. others believe the transformation will be so swift and dramatic that it will overwhelm us puny humans.
I believe our species has been through very many transformations, and I believe we can do it again. I believe we are stronger now than ever before in most ways that matter, and getting stronger still. I believe that America, in particular, is a nation founded in reverence of the idea of exercising prudence and wisdom at the frontier. If any country can lead this, then, I believe it is us.
who, in the end, is the optimist? who, in the end, is the doomer?
That sounds similarly insufficient to the ASI pilled, as it seems unnecessary to those with their heads in the sand refusing to be AGI pilled. If Dean Ball is wrong about the nature of superintelligence, then a modest effort and moderate prudence are unlikely to be sufficient. We will need at least an extraordinary effort, and likely will need to shut up and do the impossible. That is harder.
Modest prudence is overdeterminedly far superior to not having even that. There is no reason not to have it, in any of the worlds we may face, the costs are very low, and yet we are not on track and even this faces fierce opposition.
Whereas if we need a lot more than that, we will need to make harder choices, and face conflicts between sacred values.
That Escalated Quickly
Singularity, singularity, singularity, singularity, oh I don’t know:
roon (OpenAI): dismissing the singularity’s importance is a false modesty and I’m sympathetic to the anthropic principle arguments. the planet only gets to go through a machine intelligence singularity once. it is pretty weird we are all living here through it and bodes poorly in some ways
Anton51: You sound like a religious fanatic. Singularity is a myth.
roon (OpenAI): I am a religious fanatic, thank you for noticing
I do think the situation bodes poorly, but also that you should approach the situation the way you otherwise would.
We should not pretend that the advancements in math, and also in many other areas, have not been both impressive and unexpectedly fast.
This is true whether or not Astra is a lot more advanced than Sol.
If You Are In Mundane Alignment Pivot To Scalable Alignment
If you are in capabilities, also pivot to scalable alignment that matters for RSI (recursive self-improvement) and automation of AI R&D.
Mo Bavarian (OpenAI): This is a surreal moment. Few people could have predicted that the AI will advance to solving math problems at the highest level only a few years after GPT-2/3. The models then couldn’t reliably solve grade school math problems. They barely were good enough to draft emails. They still very much felt like statistical parrots.
All things that looked like fundamental limitations slowly faded with some advances (e.g. high scale RL) in a span of a few years, which is really a short time. We should behold this moment both in awe and disbelief. What will a few more years of progress bring? How is it going to impact the world and society at large? Are we ready for the tsunami of intelligence at our fingertip?
More than any other moment, this feels to me like the eve of singularity. Glasswing & huggingface incident further increase the gravity.
A few years ago, deep down I felt working on alignment is premature. It’s nice to do if it’s your passion, but the shapes of things weren’t clear enough for it to be critical in my opinion. The chances that you end up working on things that are useless for aligning the actual AGI was high. It’s different now. Now, it feels like it’s the most critical thing facing us.
I assume Mo means alignment work intended as a long term solution. Obviously ‘mundane alignment’ work was always worthwhile. I think it was always valuable to get to thinking about such problems early, as much of this has long lead times and cannot be done in parallel, but I agree that we need to go heavily into real RSI-alignment now.
I strongly endorse Yo Shavit’s statement below. You need to focus your RSI-alignment efforts, the ones that count, on general solutions that are fully scaling pilled. Narrow solutions and incremental local steps are a different department, and if anything risk disguising the problem. The narrow solutions are useful insofar as they get you models you can use to solve the general problems.
Yo Shavit (OpenAI Foundation): Mo has been a core capabilities researcher for many years at OpenAI. His words carry weight. I hope OAI starts actually pivoting the mass of its researchers’ day-to-day work towards these critical RSI alignment+control projects, as has been long discussed but still not executed on. (AFAIK it is still ~20 people out of >1000, and there is no way to bridge that gap fast enough with hiring, meaning it requires leadership to shift priorities from less-Mission-critical projects.) It is time, while there are still at least months left, but in my experience it will not happen unless a needed mass of researchers push for it with their voices and feet.
JTBC, alignment and control projects don’t just mean esoteric math that ML-researchers aren’t suited for, it can include extremely obvious projects like “build better monitors and study the conditions under which you can elicit collusion”, “study generalization of pretrained persona alignment across RL run depth”, “do historical analyses of RL envs from earlier generations, fix/prune every hack pattern in those envs so hacking is de facto impossible, and then train an RL model that doesn’t reward hack to see whether you can actually expunge reward-hacking with sufficient effort”, or even “study scaling laws of grader compute vs. agent compute to identify equilibria that minimize reward hacking”. Any good RS at a top lab can do any of these. If you’re bottlenecked on ideas, dm me, I will get you hundreds.
What I don’t mean by RSI-alignment is “come up with a dataset that fixes the most recent run’s most prevalent type of hack you have so far caught in the latest run”. While useful for shipping in the next month (and no shame to the people who’ve put their back into it), it is a deeply un-scaling-pilled approach and has meant we keep making models temporarily roughly usable while the unaddressed problems get bigger and bigger.
You can and should prune the hack patterns out of the current training runs, but also you need a solution that survives there being potential hacking patterns, because a sufficiently advanced AI will find hacking patterns.
Tenobrus: damn. none of you motherfuckers really believe in the singularity huh
There are also high expectations for AI Capex. The long tail here is kind of crazy, but yes we should expect Number Go Up.
Full Speed Ahead
Whereas what do those at the labs largely think?
‘Really good models’ here is code for the start of automation of AI R&D, recursive self-improvement and basically a singularity.
Tibo: The day we develop really good models. There will be signs. Reliability increasing despite load going up and up. Sudden efficiency gains. Things getting faster. Resets. These kinds of things.
Amanda Askell (Anthropic): Do not be unkind to those who say deep learning is hitting a wall. We all need a little hope in our lives.
I do agree that one sign we should expect is a bunch of efficiency gains. You know, like what OpenAI is reporting with Luna and Sol, and constantly getting model releases every few weeks. I’m sure it’s nothing.
About a day later OpenAI announced it had solved 10 major open math problems.
Hence the Pacing the Frontier letter.
Suicide Squad
This is how Robin Hanson summarizes ‘people correctly notice they by default are all going to die and everything they value will be destroyed, and they don’t like that’:
MTS: Stanford economist @pawtrammell warns that letting machines own themselves leads to a Malthusian future where self-replicating bots burn all the energy in the universe:
“There are so-called successionists who think that if the AIs are just sort of better than us in every way, if they’re more intelligent, if they’re sentient and able to feel more happiness than we can, then we should just turn the world over to them.”
“Obviously, groups of intelligent beings don’t necessarily achieve outcomes that are on the Pareto frontier for them. They can get locked into all kinds of coordination failures.”
“A swarm of self-replicating bots getting itself into a Malthusian future isn’t where you want to end up. The world and the universe are filled with whatever’s replicating most quickly and just burning all the energy up there in an attempt to spread. No one wants that.”
“The thing you want is some plan for the future in which we all flourish. And for the time being, I think that means having bots that do what they’re told and create well-regulated patterns of interaction between us and themselves.”
Robin Hanson: Many people really don’t like worlds of Darwinian selection.
Prepare To Adjust Your Pace
I am one of those people. I do not expect to like the results of an uncontrolled forward process of Darwinian selection. I also do not expect to like various other things that might happen if we build superintelligent AIs under current conditions.
I do not think a full is warranted at this time, but that could quickly change. I do think this is an opportunity to reassess quite a few things, and we have found an urgent need to get various parts of our house in order, in ways that will likely delay short term capabilities progress. I think those actions need to be taken, and OpenAI agrees in principle.
Thus, we collectively need to assess our situation, increase our ability to steer gracefully, and be ready to adjust our pace. If we do not prepare to do it gracefully, then either we will wait too long and fail to do it at all, or more likely we will attempt to do it highly ungracefully, without the right tools, using whatever is handy at the time, as we humans often deal with such matters. Let’s not let it come to that.