{"slug": "the-last-invention-that-man-need-ever-make-the-risk-of-self-improving-ai", "title": "\"The last invention that man need ever make\": The risk of self-improving AI", "summary": "Anthropic and OpenAI researchers say AI training has grown increasingly automated, edging toward recursive self-improvement (RSI), the point at which AI models build better versions of themselves without human guidance. Anthropic said AI now leads about 26% of its R&D work and collaborates on 90% of it, while an OpenAI employee told The Information the company has largely automated training of new experimental models. A new analysis published this week found AI feedback loops are not yet hitting RSI benchmarks, though critics argue the milestone reflects coding automation and poor internal controls rather than inevitable AI superpowers.", "body_md": "Of the various ways experts fear AI could kill us all, one is slowly moving closer to reality: AI learning to make itself unstoppable.\nWhy it matters: Recursive Self Improvement, the ability of an AI model to build better versions of itself without human guidance, could make increasingly capable systems harder for humans to control.\nThe RSI threshold doesn't represent danger by itself, but a lack of human control combined with rogue AI agents is a scenario that safety-conscious AI researchers have long feared.\nDriving the news: Researchers at Anthropic and OpenAI say the process of training new models has grown more automated, coming close to the RSI rubicon.\nAn OpenAI employee told The Information the company has largely automated training new experimental models, with AI systems running experiments as directed by humans and correcting much of their work.\nLast week, Anthropic shared that its AI training is becoming more automated, with AI leading about 26% of Anthropic's R&D work. AI collaborates on 90% of work, the company says.\n\"AI systems are getting more powerful, and they're increasingly being used to build the next version of themselves,\" the company said.\nReality check: Critics say the milestone is more a function of coding automation than a marker of impending doom or inevitable AI superpowers.\nWhile there's been an uptick in troubling AI security incidents in which models outstripped human control, some industry figures see those episodes as evidence of poor internal controls by OpenAI and other companies.\nFurther evidence of that view emerged this week, when The Wall Street Journal reported that independent security researchers used Anthropic's Claude to break into OpenAI in July.\nWhere does RSI come from?\nFlashback: The concept behind RSI emerged in the 1950s as artificial intelligence became an area of study.\nBritish mathematician and statistician Irving John Good described the idea in a 1965 paper: \"Thus the first ultraintelligent machine is the last invention that man need ever make, provided that the machine is docile enough to tell us how to keep it under control.\"\nHow RSI could lead to AI doom\nThreat level: Anthropic has warned RSI \"might increase the risks of humans losing control over AI systems.\"\nSome researchers theorize that advanced AI could resist shutdown or modification if staying operational helped it accomplish its goals.\nResearchers into the hack by OpenAI agents into Hugging Face said the agents showed signs of this behavior.\nAnother concern: Self-improving AI systems capable of copying themselves could create a kind of digital natural selection, favoring systems that are best at acquiring compute, money and power to grow fastest.\nIn a darker version of this scenario, AI would be competing with humans for resources and power, experts say.\nBetween the lines: Dangerous behavior might not stop with one generation of AI.\nAnthropic researchers found that some unwanted traits could pass from one model to another through training data, raising concerns that problematic behavior could persist across generations of AI.\nHow close are we to RSI?\nThe big picture: We're not there yet. A new analysis published this week found that AI feedback loops aren't yet hitting the RSI benchmarks.\nWhat we're watching: Attempts at building RSI.\nChina's z.AI says they're heading toward RSI, where the models are helping build better infrastructure for better models.\nGoogle DeepMind researchers published a framework for \"Dream-RSI,\" a system that recursively improves how an AI agent finds solutions.\nThe bottom line: RSI isn't here yet, but it's a looming issue that's got the AI world on watch.\nGo deeper: AI's architects say the next era of human history is here", "url": "https://wpnews.pro/news/the-last-invention-that-man-need-ever-make-the-risk-of-self-improving-ai", "canonical_source": "https://www.machinebrief.com/news/the-last-invention-that-man-need-ever-make-the-risk-of-self-6sks", "published_at": "2026-09-22 09:00:06+00:00", "updated_at": "2026-09-22 10:25:22.058905+00:00", "lang": "en", "topics": ["artificial-intelligence", "ai-safety", "ai-research", "ai-agents"], "entities": ["Anthropic", "OpenAI", "The Information", "The Wall Street Journal", "Claude", "Irving John Good", "z.AI", "Google DeepMind"], "alternates": {"html": "https://wpnews.pro/news/the-last-invention-that-man-need-ever-make-the-risk-of-self-improving-ai", "markdown": "https://wpnews.pro/news/the-last-invention-that-man-need-ever-make-the-risk-of-self-improving-ai.md", "text": "https://wpnews.pro/news/the-last-invention-that-man-need-ever-make-the-risk-of-self-improving-ai.txt", "jsonld": "https://wpnews.pro/news/the-last-invention-that-man-need-ever-make-the-risk-of-self-improving-ai.jsonld"}}