Recursive Self-Improvement: What It Would Require

A developer distinguishes four senses of self-improving AI, arguing that only in-context and scaffold-level self-modification are routine today, while weight-level and successor design remain unsolved. The post applies Amdahl's law to show that explosive recursive self-improvement requires an extremely small serial fraction, and notes that current evaluations of ML research capability are not yet settled.

“Self-improving AI” covers four quite different systems, one of which is routine today and three of which are not. Sorting them out is the difference between a testable question and a slogan. | Sense | Description | |---|---| | in-context | A system revises its own output within a session — critiquing a draft, retrying after a failed test, refining a plan. Ordinary today and bounded by the session: nothing persists, and the underlying model is unchanged. Related to self-verification and best-of-n sampling. | | scaffold | A system modifies the code around the model: prompts, tool definitions, control flow, retrieval. Persists across sessions, and the model itself is still fixed. This is where most real self-modification currently happens, and its ceiling is whatever the fixed model can do with better plumbing. | | weight-level | The system produces training data or training procedures that update its own weights. Exists in narrow forms — self-generated data, self-play in games with clean win conditions, distillation from its own filtered outputs — and does not run unattended at frontier scale. | | successor design | The system designs the architecture, training procedure and infrastructure of a more capable system. This is the sense the intelligence explosion argument requires, and no public system does it end to end. | Claims about self-improving AI are frequently supported with examples from the first two senses and used to argue about the fourth. The gap between them is not a matter of degree: an agent that rewrites its own prompt is operating inside a capability ceiling set by weights it cannot touch. Strip out the drama and the successor-design loop is a research programme with five steps: generate a hypothesis about what would make the system better; implement it; train something; evaluate it; keep the change if the evaluation improved and discard it if not. Every step is something identifiable, and each has a distinct cost profile. Two of those steps are cognitive and could plausibly be automated at speed — hypothesis generation and implementation. Two are not. Training consumes hardware for a wall-clock duration set by the size of the run. Evaluation consumes hardware too, and more importantly consumes judgement about whether the metric that improved is the metric that matters, which is exactly the failure described in benchmarks that do not transfer https://multigrid.ai/learn/benchmarks-dont-transfer . The most useful piece of arithmetic here is the oldest one in computer science. Amdahl’s law says that if a fraction p of a process can be sped up by a factor s and the rest cannot, total speedup is bounded by 1 / 1 - p + p/s . As s goes to infinity, the bound goes to 1 / 1 - p . Apply it to a research programme. If a fifth of the wall-clock time is irreducibly serial — hardware waiting, training running, results arriving — then perfect, instantaneous automation of everything else gives a fivefold speedup and no more. Getting to a hundredfold speedup requires the un-automatable fraction to be about one percent. That is not an argument that acceleration is impossible; it is an argument that the interesting quantity is the size of the serial fraction, and that a claim about explosive acceleration is implicitly a claim that the serial fraction is tiny. Two things could shrink it. Cheaper experiments, if smaller runs reliably predicted the behaviour of larger ones — which is what scaling-law extrapolation attempts and what it only partly delivers, since some behaviours appear only at scale. And better priors, if a system could tell which experiments to skip. Both are active research areas and neither is solved. There is now real measurement work here, and it is worth knowing what it does before treating any claim about it as settled. Evaluations of machine-learning research capability give systems tasks drawn from actual research engineering — improving a training script against a held-out metric, reproducing a result, debugging a pipeline — and compare their outcomes against human experts given the same task and a time budget. METR’s research-engineering evaluations are the best-known public example of the format. A second line of work measures the length of task a system can complete rather than the difficulty: how long a task would take a competent human, and how that duration scales with model generation. The appeal is that it produces a quantity with a unit rather than a benchmark percentage, and that the unit is directly relevant to whether a system could run a multi-day research programme. Both approaches share limitations that are stated openly by the people doing them, and they are worth carrying into any headline you read. The task sets are small. Scoring depends on how much scaffolding the system is given, so the same model can produce very different results. The human baseline depends on which humans and how much context they had. And the tasks are ones somebody could specify and grade, which systematically excludes the open-ended direction-setting that the successor-design loop most needs. Read the methodology before the number; in this area the methodology is the finding.