Algorithmic Progress is not Exogenous

AI safety arguments that treat algorithmic progress as an exogenous factor with a fixed 3-10x annual rate are flawed because algorithmic progress is driven by the AI ecosystem itself, according to an analysis of AI slowdown and pause debates. The author compares this to earlier 'compute overhang' arguments, which assumed compute growth was fixed even though compute growth was driven in large part by AI demand, and notes that reallocating existing fab capacity and developing AI-specific chips like those from Nvidia and AMD made gaming GPUs far less effective for training. The piece warns that assuming a constant rate of algorithmic progress repeats the same mistake.

Sometimes I have discussions about AI slowdowns and pauses where an argument against the pause is that a pause cannot slow things down in the long run since, even if we audit all large concentrations of compute, eventually algorithmic progress will lead to superintelligent AI being able to be trained on somebody’s random gaming GPU, at which point there will be a sudden massive overhang and so we will see an incredibly rapid intelligence explosion which is very dangerous. Instead, it is often argued, it is safer to actually use all of the compute for AI training so that takeoff is more continuous and, hence, theoretically, safer. A lot of this thinking relies on the idea that there is some factor called ‘algorithmic progress’ which continues to increase at 3-10x per year indefinitely making AI performance continuously better for a given level of compute. Often, it is additionally argued that since this algorithmic progress is much faster than hardware progress, then e.g. caps to the amount of concentrated hardware or AI total hardware are largely irrelevant since e.g. even if the remove 90% of the available hardware for AI training, one year later we have AIs that are 10x more efficient due to algorithmic progress and hence we have bought at most one year. The core, and very simple, point I want to make here is that arguments like this tend to assume that there is such a thing as ‘algorithmic progress’ with a fixed rate per year and crucially that it is exogenous – meaning it is not affected by other events in the industry. We saw an identical argument and failure with the notion of compute overhangs. The argument here is that it is better that frontier labs use the compute to maximally advance AI capabilities because at a given level of compute we know what the capabilities are and makes capabilities growth smoother and more predictable and ‘uses up the overhang’. Conversely, if we did not use all of the compute for AI training, the ‘compute overhang’ would keep accumulating and thus there could be a super dangerous sudden ‘spike’ of capabilities if at some point we decided to switch all of the compute from other things to AI. This argument is somewhat true but of course it assumes that compute growth is some exogenous variable that keeps occurring no matter what happens in AI, thus allowing the ‘dry tinder’ to pile up at the same rate. Of course in practice this is not what happened. Compute growth was driven in large part by AI demand , and thus attempts to reduce the ‘compute overhang’ by using the compute actually created a much worse problem and faster AI capabilities in the end<sup 1 fn:1 </sup . Importantly, some of the arguments about compute overhangs implicitly or explicitly assumed that compute growth was in fact fixed, because e.g. TSMC can only increase fab capacity slowly, which is true. Even with today’s demand TSMC is increasing fab buildout but not at a crash pace. However, it ignores the other degrees of slack the system could pull on much faster to satisfy AI demand. These include reallocation of existing fab capacity towards AI away from other uses and, crucially, developments of chips vastly better suited to AI for the same silicon area cost . Today’s AI chips from Nvidia and AMD etc have almost no relation to the original gaming GPUs that were repurposed to AI, and almost certainly AI progress would have been a lot slower if we were still trying to train models with gaming GPUs, and even if in 2030 the world somehow had a massive surplus of gaming GPUs to suddenly dump into training AI, this ‘compute overhang’ would be far, far less effective for training AI than today’s chips. I am worried that we are repeating exactly the same problems with the idea of exogenous ‘algorithmic progress’. ‘Algorithmic progress’ is not some manna descending from heaven at a constant rate<sup 2 fn:2 </sup . It is caused by the entire ecosystem of people working on AI – from AI researchers coming up with better algorithms to data companies producing data that improves AI on certain capabilities. This entire ecosystem is hugely affected by AI progress as a whole . If AI progresses rapidly, AI companies have more money, more people start to study and work on AI, the AI labs demand induces supporting ecosystems of companies, more academic research is focused onto the problems currently facing AI scaling, and more software and other supporting infrastructure is built. Now imagine the counterfactual world where AI progress and scaling is nowhere near as massive as today. Even if algorithmic progress continues to some extent – i.e. there are academics tinkering with neural nets on repurposed gaming GPUs – and suppose some academic in 2030 comes up with the ‘secret algorithm for AGI’. At this point he still has to figure out how to a. start an AI company which would be a much more novel concept both for the academic and for VCs, b. get other people to believe in this idea, c. build a huge amount of supporting scaling training and inference infrastructure to even be able to run the ‘novel AGI algorithm’ efficiently and at scale, d. somehow figure out how to network millions of scattered gaming GPUs together to train and inference the thing, and so on. Conversely, if the same thing happens today, he publishes his paper and within hours AI agents at multiple trillion dollar AI labs instantly reimplement and test it on their infrastructure. If it works, they have hundreds of thousands of super powerful specialized AI supercomputing GPUs to run it on, as well as the existing infrastructure to both scale out training of this algorithm across a massive fleet of hyper-networked GPUs, as well as the inference capabilities to instantly spin up a ‘population’ of millions of such AIs running this new algorithm. Our entire ‘reduce overhang’ strategy has thus created precisely the overhang we were worried about . We should not make the same mistake with algorithmic progress conceptually. Algorithmic progress is endogenous to the whole ecosystem of AI. Change that and ‘algorithmic progress’ can slow down or speed up. Perhaps it is helpful here to take a step back and look at what algorithmic progress actually is. Usually, it is defined as the amount of compute required to reach some level on some benchmark over time where later models require less compute. Previously, we discussed https://www.beren.io/2025-08-02-Most-Algorithmic-Progress-is-Data-Progress/ splitting the concept of ‘algorithmic progress’ in terms of ‘true algorithmic progress’ and ‘data progress’ since much of the improvements of LLMs on certain benchmarks has to do with substantially ‘higher quality’ data which is more targeted at the benchmarks themselves. Dwarkesh and co https://www.dwarkesh.com/p/pretraining-progress-is-mostly-data did a similar study in pretraining and found that a substantial fraction, but not all of algorithmic progress actually comes from better datasets vs strict improvements in architecture and training optimizers etc<sup 3 fn:3 </sup . Stepping back, the way I think about this is that what the labs have created is really a semi-automated and rapidly increasingly automated hillclimbing machine. They, or outside companies, create benchmarks that they think measure some important dimension of AI capabilities. Once this is seem as important the labs purchase data or RL environments from the ecosystem of data providers<sup 4 fn:4 </sup that are similar to the benchmark and train the capabilities required for the benchmark. Then they apply the compute to do various ablations, do various training runs and especially massive RL on the environments that train adjacent skills to the benchmark, then the model capabilities improve on this benchmark and at this skill. Obviously the labs don’t just do this for a single benchmark but a huge suite of tasks simultaneously and so we observe somewhat smooth but also very ‘spiky’ growth in capabilities when some specific skill suddenly comes into focus as a hillclimbing target. Since the labs are usually first at hillclimbing the benchmark followed by open source 1-2 generations later we thus see the typical pattern of ‘Oh my god this is amazing the models can finally do X. AGI is here ’ followed by three months later some 30B Qwen model which is OOMs smaller performing at the same level as the previous generation of frontier models. However, it is important to note that let’s suppose for a moment there is a super dramatic pause. All AI model training is banned. Now this hillclimbing infrastructure and ecosystem becomes pointless. If you are not training anything anymore there is no need for it. It disappears or goes to another country. There is no law of physics supposing that hillclimbing must continue in the absence of any training runs. Now, let’s suppose a more realistic scenario in which e.g. training runs above a certain total flop count are banned. Here things get more interesting. The entire hillclimbing ecosystem can still exist, people are still hillclimbing but with a ceiling. There are two interesting effects that happen here. Firstly, if the flop count ceiling is pretty low, this massively helps competition. The primary advantage of the frontier labs is their overwhelming amounts of compute. If they cannot use that compute for training runs, they can still use it for inference and ablations. Inference will still generate massive revenue obviously but eventually we expect that the margin obtainable will decrease, thus creating greater consumer surplus and less FCF that labs can use to continue maximizing AI progress overall. Data companies will likely continue to exist but become more specialized and focus on different verticals which are not pure RSI/coding<sup 5 fn:5 </sup . It is also important to note that there will always be some exogenous or near-exogenous AI progress occurring. This will primarily come from academia and random tinkerers which cannot meaningfully be stopped. Academics will always be doing research on novel AI algorithms and ideas and this seems exceedingly unlikely to be shut down, although it is not necessarily that dangerous either in the absence of massive compute resources<sup 6 fn:6 </sup . Indeed, academia is structurally against https://www.beren.io/2023-04-10-Why-GOFAI-failed/ bitter-lesson style scaling for the very simple reason that academia has loads of manpower and relatively little capital. Grad-student descent is much easier for them than spending hundreds of millions of dollars on GPUs. For massive hyperscalars it is the opposite. Manpower is expensive and precious; compute is cheap. For them scaling is almost the natural operation and this, I think, is why the ‘scaling laws’ took on such a potent memetic form even though if you looked at them naively they implied basically strongly sublinear returns to scale<sup 7 fn:7 </sup . Because of this while academia does generate substantial algorithmic progress, much of this is not directly useful for frontier systems and indeed often cuts away at the elegance and simplicity needed to scale. Academia is also famous for nerdsniping itself into insane and pointless subfields with minimal applicability. Academia, like most people without very strong commercial incentives, is often pretty allergic to basic data gathering, data quality, and infrastructural work compared to ‘sexier’ algorithmic innovations. Nevertheless, we can expect academia to still produce a steady stream of algorithmic progress and insight. Similarly, even in a pause world we expect industry to contribute some infrastructural and data innovations, and if our pause is primarily just stopping RSI https://www.beren.io/2026-09-08-Many-Benefits-of-AGI-Could-Still-Be-Realized-In-A-Pause/ , then this will primarily not necessarily slow down algorithmic progress deeply compared to today although likely compared to the full-RSI counterfactual but rather redirect it towards non-RSI areas with only some leakage of general algorithmic progress which transfers directly to RSI. The picture we thus come to is mixed whereby we can think of algorithmic progress as partially exogenous but heavily endogenous. Not 100% of algorithmic progress can be stopped without extreme measures, but nevertheless, it seems likely that it can be substantially slowed through regulatory changes in AI, in the same way that scientific progress in other potentially dangerous fields such as biological and nuclear weapons has been slowed by regulation and arms treaties. More broadly, current AI is a positive feedback loop – progress attracts capital, revenue, talent – which results in more progress. If one input to the feedback loop is stopped, the rest of the loop does not just keep looping exogenously. Rather, depressing any of the inputs depresses the others. Because of this, the problem is easier to solve than some assume but nevertheless, achieving any kind of pause or slowdown through regulation does not require reversing a lot of momentum that has built in the industry as well as overturning the short term vested interests of many who have wealth and political power through investments or jobs in the current AI industry. 1. In general this is a classic case of pessimization that Richard Ngo discussed https://www.mindthefuture.info/p/on-pessimization , and really just impressively shows how much power and economic incentives can reshape argument structures even inside your own mind . ↩ fnref:1 2. A lot of this issue comes from applying classic economic marginalist thinking meant for regimes of perfect competition to regimes which are extremely far from perfect competition where indeed there are only a few extremely concentrated players who support an entire ecosystem. The key to marginalist thinking is that it lets you simplify the situation by assuming that the consequences upon others of your own actions are negligible. This may be roughly true if you are a small seller of a commodity in a market with millions of sellers of the same commodity. It is not true if you are one of <10 companies pursuing frontier AI in the entire world and shaping capital streams of substantial fractions of world GDP. ↩ fnref:2 3. To Ryan Greenblatt’s point https://www.dwarkesh.com/p/ryan-greenblatt , obviously a lot of things we think of as ‘data progress’ are also algorithmic. Knowing how to filter common crawl well, how to setup good synthetic generation loops, and how to design rubrics for human task completers, all of these are ‘algorithmic’ tasks in some sense. But at the same time, I think there is still an important distinction here. ↩ fnref:3 4. There is now a fascinating trend of some academic group coming up with a benchmark that gets traction on AA or someplace, then spinning out a data company to sell data or RL environments that lets labs hillclimb this precise benchmark ↩ fnref:4 5. In fact, any kind of pause will greatly extend the life of data companies which in the RSI takeoff world only exist in the short window before the RSI loop is closed. A full AGI can explore the world and obtain the data it needs for itself autonomously without intermediation of a separate human-run data company. Fascinatingly this is another example of people not being properly AGI-pilled. Almost all of the companies that provide inputs to AGI labs will be rendered obsolete by an RSI-ing lab. THe AGI can obtain its own data, build its own compute, doesn’t need AI researchers or AI investors etc. Literally everybody maybe a few execs in the RSI-ing AI lab have a strong interest in avoiding strong RSI. It is simply the coordination problem that is the challenge. ↩ fnref:5 6. It is worth thinking here about the comparison to certain fields of biotech. For instance, human genetic engineering and creation of bioweapon viruses are both heavily banned in international treaties with nontrivial teeth including China . Nevertheless, some academics continue to nibble around the edges of these areas and make progress here which may eventually spill over uncontrollably. At the same time, the situation here is vastly more stable than if we had multiple trillion dollar companies each competing to make most eugenic superbabies or the most lethal viruses. ↩ fnref:6 7. In the time before AI, ‘scaling’ and ‘scale’ were already highly prestigious terms in Silicon valley. YC was teaching courses on ‘blitz-scaling’ companies, and Google engineers were famous for boasting about the immense scale they were operating at on hackernews threads. AI scaling laws were perfectly memetically engineered to fit into this landscape and worked incredibly well by doing so. ↩ fnref:7