Many Benefits Of AGI Could Still Be Realized In A Pause

A well-targeted AI pause focused narrowly on recursive self-improvement (RSI) and related capabilities could increase the rate of non-AGI technological advancement in the short run, according to an analysis of AI race justifications. The analysis argues that only a narrow slice of AI is deeply dangerous — AI applied to RSI, AI research, hardware design, and AI-related coding — while uses such as coding assistants for SaaS apps, AI-assisted mathematical proofs, and systems like AlphaFold for protein folding are safe. Because AI progress is spiky and driven by specific datasets and RL environments, redirecting the resources labs currently pour into automated AI research and long-horizon coding toward domains such as biology and materials science would increase progress there while improving safety.

Often I hear justifications for continuing the AI race that even if there are dangers, there are also massive technological benefits waiting if we race ahead and successfully align AGI. For instance, massively improved medical care such as treatments for cancer, aging etc. Solving much of science and math and the resulting technological benefits, and so on. These benefits are then asserted to be so great that it is perhaps worth accepting some nontrivial risk of extinction to realize them sooner<sup 1 fn:1 </sup . Conversely, it is often assumed that a serious AI pause would dramatically slow these effects. This is true to some extent, and certainly will be true in the long run if superintelligence actually is all it’s cracked up to be. However, in the short run things are definitely more complex and ironically a pause, if implemented well, could even increase the rate of non-AGI technological advancement. The reason is very simple. An AI pause does not need to blanket ban all AI. Only a very narrow slice of AI is deeply dangerous, which is AI applied to RSI and related fields such as AI research, hardware design, AI-related coding, and so on. Pretty much everything else is ‘safe’. People using coding assistants to design SAAS apps is safe. People using AI to prove mathematical conjectures is safe. People using AI systems like alphafold and descendants to crack protein folding is safe. Basically all scientific discovery is completely safe. Moreover, far from there being some super generic ‘intelligence factor’, as with humans, AI progress is intensely spiky and driven hugely by the specific datasets and RL environments the models are trained upon. These data and environments obviously cost money and human and AI labour to construct, so there is obviously opportunity cost to choosing some domain of data vs others. Right now, of course, the labs are focusing on the most dangerous possible approach and pouring massive resources and data into automated AI research, long horizon coding, and explicitly shooting for RSI. However, a well-targeted pause would need to pause only these precise capabilities, not AI capabilities in general. This means that a pause, if implemented well, could actually free up the immense resources that the labs are currently focusing on strong RSI towards other technological domains such as biology, materials science etc, which would presumably increase progress here. This means that in the short term, a pause could actually increase broader diffusion and technological benefits from AI while also massively increasing safety. In the long run, of course, preventing RSI means that our AI systems become generally less intelligent; thus the pace of progress in other fields will slowly drop below the counterfactual ‘full-speed ahead’ RSI future. However the spikiness of AI intelligence might mean that this process takes a fair bit longer than you might naively expect. Presumably during RSI almost all resources will be focused on continuing RSI, including all compute, meaning effectively that even in an ‘aligned’ RSI future, essentially the entire world is shut out of obtaining any nontrivial amount of compute as discussed in the recent Dwarkesh podcast https://www.dwarkesh.com/p/dylan-patel-3 , since the labs can effectively outbid the rest of the world for access to compute for accelerating RSI. Beyond the extremely obvious massive concentration of power concerns arising from this, this hardly seems conducive to broadly beneficial scientific progress and technological diffusion. To implement this in practice, a pause need not ban all ‘AI progress’ broadly defined but rather very specific AI progress focused on RSI. That is increasing AI capabilities at increasing models capabilities further, AI research into core remaining bottlenecks to AGI, and more generally deeply long-horizon environments teaching general strategic takeover-relevant skills. Very importantly this should not prohibit specialist AI systems and RL environments in non-RSI relevant domains such as math, biology, applied physics/chemistry, social science and so on. If through regulation the industry can be moved towards a world of systems like alphafold and specialist theorem-provers rather than hyper-general RSI-focused LLMs everybody is focused on today. The downside of this pause is that it is much harder to enforce. A ‘shut it all down’ style pause focused simply on preventing large compute buildouts is both very hard to game and secondly eliminates the physical substrate for the massive training runs required to advance frontier AI. This more precise focus on preventing general RSI-capable and focused AGI but instead designing specialist AIs for non-ML fields is much harder to enforce since it requires continuous monitoring on what exact datasets, environments etc make it into big training runs. Moreover, it introduces much more gray area where people can argue e.g. that ML research environments perhaps transfer to biology and other science research and hence they should be included and then you ‘accidentally’ end up with an RSI-capable AI anyway. Nevertheless, even if the pause includes strong caps on total training compute, there is nevertheless massive gains to be had by specialization, because transfer learning is not that powerful. Imagine taking the entire compute and dataset budget of a contemporary frontier model and focus all of that on a narrow goal such as advancing biology research. LIkely there is a factor of 10-100 in biology-specific capabilities to be gained even with fixed total compute. Moreover, this ‘biology model’ will be extremely powerful in its domain, but seems very unlikely that it will spontaneously generalize to RSI. This appears to be not how current deep learning works. The biology model is thus extremely safe while being capable of driving huge amounts of progress in biological research to get us the cures for cancer, aging etc that we have been promised by full AGI. There will still be some dangers here. For instance, people could misuse the biology model to produce super viruses and in general it will accelerate biowarfare capabilities. However, this is not a unique AI danger but is just a very general danger of all technology. Google search dramatically increased people’s access to biowarfare relevant knowledge also. This kind of danger feels much more likely to be able to be handled by existing legal mechanisms and institutions than full-fledged recursively self-improving AGI. Because of this, likely the next few OOMs of AI progress in non-RSI related fields can be obtained safely with carefully designed Pause regulation while remaining very safe. However, the key challenge here is necessarily enforcement of such a fine-grained pause, while enforcement is already a massive challenge even with crude proposals. It is important to note however that this RSI-only pause might actually be easier to enforce in some sense since it provides an alternative business model to existing AI labs. Rather than just essentially destroying their entire business outright, the regulation says they can continue being AI companies, they are simply banned from pursuing RSI but can still monetize their amazing AI capabilities in all other domains. This removes perhaps the strongest reason for AI labs to defect on the pause, namely that they would cease to exist otherwise. Such a pause would turn existing AGI labs into still fabulously wealthy and powerful companies but at the level of a ‘super google’ rather than an uncontrolled entity that eats the entire economy and ultimately the light cone. 1. For instance, from a purely selfish perspective, we each have some nontrivial probability of death every year and that this probability rises exponentially https://en.wikipedia.org/wiki/Gompertz–Makeham law of mortality with age. If you only cared about your personal survival your willingness to risk AGI to produce anti-aging technology should rise proportionately. ↩ fnref:1