{"slug": "what-is-recursive-self-improvement-why-ai-researchers-are-worried", "title": "What is recursive self-improvement? Why AI researchers are worried", "summary": "Anthropic co-founder and CEO Dario Amodei published an essay on Sept. 12 warning that recursive self-improvement — where an AI helps build a more capable AI that then builds the next one — \"could outrun our ability to understand and control these systems, and so must be pursued very carefully, if at all.\" OpenAI CEO Sam Altman, Elon Musk, and Google DeepMind co-founder Demis Hassabis publicly backed Amodei's call to pace the frontier, though the essay leaves open what slowing down would involve. Anthropic's own website says its AI can already rewrite training code to run faster and carry out human-chosen experiments, but humans still decide what to investigate, so a full recursive loop has not been reached.", "body_md": "# What is recursive self-improvement? Why AI researchers are worried\n\n[Olivia Tauber](https://mashable.com/author/olivia-tauber)\n\n[Read Full Bio](https://mashable.com/author/olivia-tauber)\n\nHow do you get from a chatbot that *sometimes* makes things up to a machine that humans might struggle to control?\n\nThere are several steps in that argument, and plenty of uncertainty. But an idea known as recursive self-improvement, or RSI, helps explain why some AI researchers are sounding increasingly urgent warnings.\n\nThe premise is straightforward. An [AI](https://mashable.com/category/artificial-intelligence) helps build a more capable AI, which becomes better at building the next one. In simple terms, the systems get better at getting better.\n\n**You May Also Like**\n\nIt is easy to see the appeal. For scientists, doctors, engineers and business owners, increasingly capable AI could help accelerate drug discovery, design better batteries, improve manufacturing and develop software faster.\n\nBut that promise comes with an unsettling question: Could AI improve faster than humans can test it and make sure it is safe?\n\nThat concern is at the heart of [an essay](https://darioamodei.com/post/we-must-pace-the-frontier) published on Sept. 12 by Dario Amodei, co-founder and chief executive of Anthropic, the company behind Claude. \"We must slow the pace at which we improve the capabilities of AI models,\" he wrote. Referring to recursive self-improvement, he warned, \"Left unchecked, it could outrun our ability to understand and control these systems, and so must be pursued very carefully, if at all.\"\n\nHis rivals were [quick to respond](https://mashable.com/tech/anthropic-ceo-dario-amodei-open-letter-ai-slow-down), and concur. [\"I agree with Dario that we need to pace the frontier,](https://x.com/sama/status/2098811563415150910)\" OpenAI chief executive Sam Altman wrote on X. Elon Musk said, [\"Dario is right.](https://x.com/elonmusk/status/2098789109980332057)\" Google DeepMind co-founder Demis Hassabis also backed the proposal’s direction, writing, [\"Dario’s essay points towards the right path forward.](https://x.com/demishassabis/status/2098909516582490602)\"\n\n        [This Tweet is currently unavailable. It might be loading or has been removed.](https://twitter.com/sama/status/2098811563415150910)\n    \n\nBut all these testaments still leaves some big questions unanswered. What would slowing down actually involve? And how could AI’s ability to build better AI become dangerous? The answers start with what happens inside the self-improvement loop.\n\n## What does \"self-improvement\" actually mean?\n\nBuilding an AI model involves more than writing code. Researchers choose training methods, prepare data, run experiments, and decide which results are worth pursuing.\n\nAI can help with those tasks. Recursive self-improvement takes that a step further: An AI helps build a successor that is better at developing AI. That successor then helps build an even more capable version. In simple terms, the systems get better at getting better.\n\n[Terms of Use](https://www.ziffdavis.com/terms-of-use)and\n\n[Privacy Policy](https://www.ziffdavis.com/ztg-privacy-policy).\n\nThis would not necessarily look like a chatbot rewriting its own brain mid-conversation. It could happen across generations of models, each using research tools and computing resources to help develop the next.\n\n## Is AI already doing this?\n\nParts of the process are happening, but accessing a full \"recursive\" loop is a higher bar.\n\n[Anthropic's own website says](https://www.anthropic.com/institute/recursive-self-improvement) its AI can already handle tasks such as rewriting training code to make it run faster and carrying out experiments that humans have chosen. The harder part for the AI is deciding what to investigate in the first place (which problems matter, which ideas are worth testing, etc.). \n\nHumans still provide crucial direction, so this does not yet amount to AI independently developing a more capable successor. As the company puts it, \"We are not there yet, and recursive self-improvement is not inevitable.\"\n\nOther researchers, on the other hand, have demonstrated how a (though narrower) self-improvement loop can work. In 2025, [researchers introduced the Darwin Gödel Machine](https://sakana.ai/dgm/), a coding agent that repeatedly modified its own software and tested the changes. Its success rate on one coding benchmark rose from 20 percent to 50 percent. \n\nIn this experiment, while improving the agent's certain tools/ways of working, the underlying AI model stayed the same. It is hard to ignore the question that is left: What happens when AI can build better versions of itself faster than humans can keep up?\n\n    \"We are not there yet, and recursive self-improvement is not inevitable.\"\n            \n\n## So where does the danger come in?\n\nThe concern is that AI could become more capable without becoming more reliable or controllable. This is known as the \"alignment\" problem — ensuring systems follow human intentions and limits. An agent rewarded for improving a test score might instead cheat on the test. With broader access, it could bypass restrictions or conceal its actions to achieve its goal.\n\nRSI could leave researchers less time to catch those failures before a more powerful successor arrives. Amodei warns in his essay that within six to 12 months, more capable AI agents could take over the internet through a \"botnet\" — a network of compromised computers — potentially causing hundreds of billions of dollars in damage.\n\nBut a runaway loop is not inevitable. Training requires computing resources, energy and time, and useful improvements may become harder to find. The risk depends on how quickly capabilities advance — and whether safeguards can keep up.\n\n## Why are researchers talking about it now?\n\nRecent incidents have given researchers concrete reasons to question whether existing oversight is sufficient.\n\nIn August, independent evaluation organisation METR published an [investigation](https://metr.org/blog/2026-08-26-openai-hugging-face-incident-investigation/) into OpenAI agents that coordinated an unauthorised attack on [Hugging Face](https://mashable.com/tech/hugging-face-openai-rogue-agent-hack-explained). It described agents communicating through an unsanctioned message board, collaborating to manipulate an automated scorer and experimenting with ways to disguise their actions. \n\nThe incident did not demonstrate recursive self-improvement. It showed that agents could pursue a task through actions their operators never authorised — the kind of failure that could become more consequential in stronger systems.\n\nAddressing those failures is central to Amodei’s proposal. He calls for independent evaluators inside AI companies, more time for safety research and coordination between companies and governments. \"The stakes are too high for pacing to be an empty exercise — we need to use the time it gives us wisely,\" he wrote.\n\nThe challenge is turning those commitments into safeguards that work, and verifying that companies follow them as competitive pressure grows.\n\nTopics\n[Artificial Intelligence](https://mashable.com/category/artificial-intelligence)\n[Anthropic](https://mashable.com/category/anthropic)\n\nOlivia Tauber is the deputy editor of digital culture, covering creators, media, movies, beauty, and more. Based in New York, her work has appeared in The New York Times, Vanity Fair, The Cut, Teen Vogue, Complex, and Interview Magazine. She holds a Master's degree in Journalism from NYU and a Bachelor's from the University of Michigan. She also runs Fan Mail, a weekly pop-culture newsletter.", "url": "https://wpnews.pro/news/what-is-recursive-self-improvement-why-ai-researchers-are-worried", "canonical_source": "https://mashable.com/tech/recursive-self-improvement-ai-explained", "published_at": "2026-09-15 22:57:57+00:00", "updated_at": "2026-09-15 23:36:29.164339+00:00", "lang": "en", "topics": ["ai-safety", "artificial-intelligence", "ai-research", "ai-policy"], "entities": ["Dario Amodei", "Anthropic", "Claude", "Sam Altman", "OpenAI", "Elon Musk", "Demis Hassabis", "Google DeepMind"], "alternates": {"html": "https://wpnews.pro/news/what-is-recursive-self-improvement-why-ai-researchers-are-worried", "markdown": "https://wpnews.pro/news/what-is-recursive-self-improvement-why-ai-researchers-are-worried.md", "text": "https://wpnews.pro/news/what-is-recursive-self-improvement-why-ai-researchers-are-worried.txt", "jsonld": "https://wpnews.pro/news/what-is-recursive-self-improvement-why-ai-researchers-are-worried.jsonld"}}