{"slug": "making-sense-of-recursive-self-improvement", "title": "Making sense of Recursive Self Improvement", "summary": "A METR report found that the rate of progress in AI R&D at Anthropic has roughly doubled since October 2025, with most of the acceleration attributed to Anthropic's own use of AI in R&D. The analysis argues a crude form of recursive self-improvement is already underway, driven largely by AI agents speeding up the RL data-generation pipeline—interviewing experts, designing simulated environments, and verifying rollout outputs—rather than by fully autonomous model contributions. It also notes the report redacted key datapoints at Anthropic's request and that Anthropic is preparing to go public.", "body_md": "There’s a whiff of an imminent RSI in the air. The theoretical possibility of AIs recursively improving themselves autonomously until we hit singularity has always existed but the recent METR report seems to have spooked a lot of rationalists and AI insiders about how soon it can arrive.\n\nThe report that triggered the [fear](https://x.com/DKokotajlo/status/2120204075803500595?a=zawk5t8LkdCGXis_cUg&s=20) and [paranoia](https://www.lesswrong.com/posts/biFHYdJnMSCmW9Xou/orienting-to-1-year-agi-timelines) is [this one](https://metr.org/blog/2026-11-05-anthropic-ai-rd-acceleration-assessment/#executive-summary), and the chart to pay attention to is the following:\n\nIt effectively shows how **the** **rate** **of progress in AI R&D has roughly doubled at Anthropic since October 2025, and METR attributes most of the acceleration to Anthropic’s own use of AI in R&D**. Before moving on to making sense of what’s driving this and its potential implications, keep two caveats in mind:\n\n1. This report redacted many important datapoints as Anthropic didn’t let METR share them\n2. Anthropic is soon going public, so it is in their best interest to project an acceleration (Keep this in mind for the other Anthropic charts below too: they’re from their IPO roadshows)\n\nDespite these caveats, I will make the case that a crude form of recursive improvement in AI models is already happening and it’s simply going to become more sophisticated and significantly scaled up. And whatever we call ASI at the end of it is likely to be jagged.\n\n## Current self-improvement: AI models accelerating AI models releases\n\nWell, [models are releasing faster lately](https://www.reddit.com/r/ArtificialInteligence/comments/1wu1t90/new_models_used_to_come_out_every_10_weeks_now/). There are several factors driving it and perhaps the biggest one is the [sprawling data-generation industry that’s $10bn now, growing rapidly and perhaps could be $100bn in a few years](https://latent.space/p/handshake/). These data generation companies make RL environments to sell to labs on as many economically useful tasks as they can codify. \n\nWhat does a typical RL environment look like? Imagine slicing the entire economy into sectors and then sub-sectors and sub-sub-sub sectors, and then hiring an expert to design context, task and success rubrics for each one of them. As an example, imagine *Finance → Private Equity → Leveraged Buyout → Crossborder LBO* as one of the workflows needed to be automated. To automate this, a data generation company would hire an actual expert at it who’d design a simulated but faithful work environment for an agent comprising of data room, Excel sheets, and success criteria. Frontier labs buy these environments by the thousands and do rollouts in their post-training in order for the agent to get better at this task.\n\nThe RL-driven improvement has been known for years, but **what’s changing rapidly is how AI agents themselves are improving all parts of the RL process**. Consider how currently capable AI agents could help speed up the process:\n\n- Interviewing the human experts to elicit their knowledge \n  - Asking them about edge cases, finding inconsistencies, suggesting rubrics, etc.\n- Designing realistic virtual environments for RL to mimic an actual deployment \n  - Designing Slack replicas, simulating internet access, and so on.\n- [Agentically verifying outputs of RL rollouts](https://joinhandshake.com/research/ai/gandalf-the-grader/) by integrating multiple signals\n  - Figuring out whether the agent produced the hundred things it was asked to do\n\nThe upshot of this process is really that today’s agents bring efficiencies that make tomorrow’s agents better, cheaper, faster and arrive sooner. METR found the same at Anthropic: R&D is accelerated mainly through AI assistance rather than fully autonomous contributions. That is exactly the crude form of recursive improvement I mean. (We can call it ‘crude RSI’).\n\nThe proof has started arriving about acceleration in recursive improvement in AI R&D. Here’s Anthropic showing how their pretraining efficiency has improved (Y-axis is log scale, so the upward bend is the acceleration):\n\nAnd here’s Anthropic talking about how Claude Mythos got Claude running on AMD chips in about 3 days versus 6–12 months the traditional way.\n\nAnd here are more data points on AI helping improve AI (via [Dwarkesh](https://www.dwarkesh.com/p/rsi)):\n\nAs you can see in the graphic above, AI-led R&D work is increasing rapidly at AI labs.\n\n## The chasm of verifiability\n\nMany now agree that AIs are probably already better at coding and math than most professionals. The driving force behind this is the fact that these domains are cheap to verify and closed.\n\nBut the world consists of domains that are hard to verify. These are the domains where feedback is either noisy, costly or takes a long time to arrive. As I had mentioned in [What domains will be the last ones to get automated from AI?](https://invertedpassion.substack.com/p/what-domains-will-be-the-last-ones), one can visualize this as a 2x2:\n\nMany skeptics argue that progress in math and coding does not automatically transfer to hard-to-verify domains. I think there are two answers to the skeptic:\n\n1. An AI doesn’t have to solve a messy domain, it just has to be better at it than a human (and then [we can replicate such AI infinitely many times](https://invertedpassion.substack.com/p/why-frontier-labs-are-scaling-pilled) )\n2. Meta-skills learned from verifiable domains may transfer to other domains\n\n## Beating humans at non-verifiable tasks\n\nThe way to think about what AI does while in training is to juice out whatever signal is there to juice out from the data. **If there’s a pattern to be mined, gradient descent will eventually find it and learn it**. This is what makes [LLMs less about language and more about patterns manifested in language](https://invertedpassion.substack.com/p/llm-are-universal-simulators).\n\nThe easy-to-verify domains (such as math and coding) are pure signals, and hence have been the first to fall to automation. But harder-to-verify domains are not pure noise; they contain *some* signal. Evidently, humans do learn signals in them. You have great writers, entrepreneurs and investors. The entire $10bn (and growing) **data industry is hell-bent on capturing their expertise and distilling into environments that agents can learn from**. Economic forces suggest that whatever signals could be mined from economically valuable domains will eventually be mined.\n\nAnother reason why we should expect models to get better at hard-to-verify domains is that many meta-skills gained while training on verifiable domains could also transfer to other domains. Presumably, OpenAI would not have had an environment containing human-written solutions to Millennium Prize problems and yet they [solved Navier-Stokes](https://openai.com/index/navier-stokes-solution/). That is still math, the most verifiable domain there is, but it is a clear example of out-of-training generalization. It would have happened because training would have instilled in the model skills such as:\n\n- Persistence\n- Connecting distant fields\n- Error correction\n- Exploration\n- And so on..\n\nNow when you take a trained model with such meta-skills and at deployment-time give it an impossible problem, it will utilize the particular context of the problem and all the compute you throw at it to do things it was never trained to do.\n\nThere’s another subtle but important point that people tend to miss. There are some domains which behave [anti-inductively](https://ankitmaloo.com/anti-inductive/) or exhibit non-stationarity. That is, these are the domains where the “correct” answer is not fixed and, in fact, once you know a good solution, it tends to not remain good. Think of trading ideas in the stock market or fashion trends or UI styles. \n\nSurely, AI would not be good at such domains because what does being good even mean here?\n\nWell,\n\na) **The goal is not to “solve” such non-stationary domains, but simply to be better than any human agent out there**. You cannot perfectly predict weather for years in advance. Some problems are simply difficult, even for an entity with near-infinite intelligence. But could you do better than the status quo? Yes! We have Jane Street beating a random day-trader, and tomorrow’s AI could beat today’s Jane Street.\n\nb) **The AI could pick meta-patterns of exploring, adjusting and iterating just like humans do**. If a field exhibits evolution based on participants, AI could learn to monitor the existing state of the field and condition its response based on its findings. I bet [frontier labs are training their models on a fixed historical corpus and figuring out how they can predict the subsequent breakthroughs](https://www.ycombinator.com/library/P3-demis-hassabis-agents-agi-the-next-big-scientific-breakthrough) that actually happened.\n\n## RSI-driven intelligence explosion, and why it will be jagged\n\nI do think we’re going through [some version of software-led singularity](https://www.dwarkesh.com/p/rsi). By this I mean two things:\n\n1. Software automation is improving rapidly, and I fully expect superhuman AI coders and then AI researchers within a few years\n2. The improvements from skills that make AIs superhuman at easy-to-verify domains will spill over to all the rest of the domains\n\nI’m very confident about the first point and somewhat confident about the second point.\n\nAs AI-led R&D work keeps increasing at AI labs, we’d soon have superhuman fully automated AI researchers who will day and night try squeezing whatever signal they can squeeze. This includes activities like:\n\n- Improving sample efficiency, so you learn more per data point\n- Automatically constructing thousands of RL environments for future agents to learn from\n- Learning how to utilize scarce expert input *most* efficiently so it doesn’t become a bottleneck\n- Building datasets with entire histories being held out to develop algorithms that can learn meta-patterns in how institutions or entire domains change\n- Building realistic games to learn how to extract even minor signals from complex confounds\n- Utilizing progress in math to apply abstractions to other domains (e.g. what happens when you apply topology to poetry?)\n\nWill these gains in verifiable domains transfer to hard-to-verify domains? Well, they already have, but unevenly:\n\nNotice the ordering in the bottom panel: designing and reading experiments is racing ahead, judging research proposals lags, and proposing the ideas a field pursues next is at 18% of human level. The harder it is to verify, the further behind.\n\nTo reiterate, what’s driving this transfer is following main factors:\n\n1. Economic incentives to codify more-and-more of expertise into RL environments\n2. Transfer of meta-skills from verifiable domains (like quick adaptation to a domain or iterating based on feedback)\n3. Better infrastructure for agents at deployment time (more exploration, longer “thinking”, better tools and so on)\n\nOne intuition pump for the possibility of skill generalisation across domains is us humans. **Our ancestral environment of hunting and gathering never prepared us for modern economy,** and yet we’re able to apply the same meta-skills of persistence and learning from each other to build spaceships and LLMs.\n\nIt looks like to me the whatever we call ASI is likely to be jagged with unfathomably large improvements over the best human in verifiable domains and only fathomably large improvements in hard-to-verify domains.\n\n## What happens once we have ASIs\n\nWell, nobody can tell. Conditional on no AI slowdown, I expect us to have AIs that:\n\n- **Are better** at all domains than the best human\n- **Are numerous**\n- **Are radically faster** than any human out there\n\n**Once we have ASIs, the central question will be of control**. That is:\n\na) Are superintelligences controllable in a predictable fashion?\n\nb) If so, who controls them?\n\nLet’s look at both these scenarios.\n\n### The default scenario: uncontrollable ASIs\n\nThe rationalist community has been talking about the danger of misalignment and control for over a decade now. The central issue is that AIs are trained to be successful at tasks given to them, and they might go to great lengths to be successful at the task causing all sorts of collateral damage. This basic worry is exemplified by the paperclip maximizer: you ask an advanced AI to get you some paperclips and it converts the entire Earth into a paperclip production factory.\n\nAll of this seems far-fetched and so out there that it borders on being a laughable caricature. But we saw preview of something like this with the recent [OpenAI-Hugging Face incident](https://metr.org/blog/2026-08-26-openai-hugging-face-incident-investigation/) where agent swarms broke out of their test environment and compromised Hugging Face’s production infrastructure to cheat on the task given to them.\n\nNotice that this level of sophistication was exhibited by current models, which aren’t ASIs. And notice the jaggedness: **the agents were superhuman at the verifiable part of the job (finding and chaining exploits) and poor at the hard-to-verify part (what was actually wanted of them)**.\n\nOne worry I have is that even if we have rapid improvements *only* in highly-verifiable domains such as software, things could go really bad. We’re highly dependent on our cyberinfrastructure and an ASI taking over our digital infrastructure can wreak immense damage. Again, we’ve started seeing a preview of this in the increasing number of cyberattacks. (And that, too, with our current model capability levels). I urge you to skim through [AI2027](https://ai-2027.com/) for a concrete scenario on how things could unfold even if we only have rapid progress on verifiable domains.\n\nWhat could happen when we have models 10x smarter in the next few years? There’s extreme uncertainty about the exact scenario, but things could go really bad and could include loss of human life or even human extinction. [In an earlier essay](https://invertedpassion.com/human-behavior-is-an-intuition-pump-for-ai-risk/), I wrote how humanity pursues its own goals with little care about collateral damage to lesser intelligences. It’s our intelligence and collaboration that enabled us to dominate and command Earth’s resources. So if an entity much smarter than us comes along which we don’t control, it would pursue its own goals, wouldn’t it?\n\n### Controllable ASIs, but who controls them?\n\nIn this scenario, we solve the control and alignment issue of ASIs and we now have a powerful entity that listens to our commands. The future unfolds depending on who controls them:\n\n1. **The bad guys control ASIs** : imagine rogue nations, terrorist organizations, or dictators have ASIs. They could wreak significant damage to the world. I’m less confident in this scenario because running models requires chips and they might get harder to procure for rogue entities. But future models could require less compute and motivated entities could figure out creative ways to evade sanctions.\n2. **The good guys control ASIs but they become bad guys over time** : well, as the saying goes, “power corrupts, and absolute power corrupts absolutely”.  Nations could become more autocratic via constant surveillance, or frontier labs could deny model access to organizations that don’t align with their value systems, or some nations could be permanently left behind. Even if we get to UBI, in this scenario,[most humans become totally dependent on the governments, concentrating their power](https://invertedpassion.com/are-we-headed-towards-a-stable-dystopia/) .\n3. **The good guys control ASIs but gradually give up on that control** : this scenario of “[gradual disempowerment](https://gradual-disempowerment.ai/) ” suggests that AIs get so good that we delegate our economy, governance and judiciary to them, leaving humans no power over future outcomes. ASIs then slowly transform the world in their own (weird) image.\n4. **The good guys control ASIs and remain good guys** : this is the scenario that’s fuelling the AI industry. Solving cancer, more material progress, happy times for everyone. Of course, this scenario is conditional on redistribution of newly generated wealth from AI, and we should NOT take that for granted. Humanity would have to fight for it.\n\nAs you can see above, many scenarios turn out badly for the majority of people. So let’s try expanding on the 4th scenario, which is the *only* positive scenario in the list. It’s highly undersketched, but here it is.\n\n### Unbundling what jobs give us\n\nThe implications of having (controllable) ASIs in the hands of good guys who remain good guys inevitably lead us to the question of jobs. If we keep insisting on humans having a job, we’d create a bottleneck because an ASI could have done that job faster, cheaper and better. **If ASI is jagged, the jobs that last longest will sit in the hard-to-verify corner of the 2x2**.\n\nBut the capability of AIs to do *all* the jobs is rapidly increasing already.\n\nWhile the potential to do a job exists, there’s significant lag in the real economy as the speed of change is dictated by the slowest bottleneck (see the [O-ring model of economy](https://en.wikipedia.org/wiki/O-ring_theory_of_economic_development)). But sooner or later, things do change. We migrated our economy to the Internet, then mobile and now adoption of AI has begun all around us. So we’d have to confront the question of jobs soon enough.\n\nMy personal take on jobs is that we ought to unbundle what jobs provide to people.  What people want is money foremost and then [collateral benefits](https://arxiv.org/abs/2609.11019) like time-structure, meaning and social connections. **Wouldn’t it be wonderful to unbundle these and seek ways to satisfy** **all** **of them?**\n\nGiven the rapidly increasing abundance due to ASIs, we should figure out the money part via redistribution and derive other benefits from made-up projects involving humans. Think of volunteering projects, music concerts, sports competitions, conferences, and so on.\n\nIn the positive utopian vision of an ASI future, humans should ready themselves for a [ludic life](https://www.researchgate.net/publication/333560551_In_Defence_of_the_Post-Work_Future_Withdrawal_and_the_Ludic_Life). We should play, [gossip](https://meaningness.substack.com/p/exodus-from-the-circumrational-shantytown) and march for more UBI while machines create all the abundance for us to enjoy (until we collectively decide to give legal selfhood to machines, and at that point, we have no choice but to upgrade ourselves and co-birth strange but wonderful hybrids with AI).\n\nIn my gut, I can’t believe this is happening or will happen. It’s an utterly strange feeling to look outside the window and see most of the world going about its business as if all of this is simply stuff of a science fiction book.\n\nYet, I can’t avoid believing this will happen within my lifetime, and perhaps within the next few years.", "url": "https://wpnews.pro/news/making-sense-of-recursive-self-improvement", "canonical_source": "https://invertedpassion.substack.com/p/making-sense-of-recursive-self-improvement", "published_at": "2026-11-17 15:02:56+00:00", "updated_at": "2026-11-17 19:56:54.998049+00:00", "lang": "en", "topics": ["artificial-intelligence", "ai-research", "ai-safety", "ai-agents", "machine-learning"], "entities": ["METR", "Anthropic", "Daniel Kokotajlo", "LessWrong", "Handshake"], "also_reported_by": [], "alternates": {"html": "https://wpnews.pro/news/making-sense-of-recursive-self-improvement", "markdown": "https://wpnews.pro/news/making-sense-of-recursive-self-improvement.md", "text": "https://wpnews.pro/news/making-sense-of-recursive-self-improvement.txt", "jsonld": "https://wpnews.pro/news/making-sense-of-recursive-self-improvement.jsonld"}}