Anthropic's RSI warning contrasts with IPO filing

Anthropic published research Thursday warning that AI systems capable of recursively improving themselves could arrive sooner than most institutions are prepared for, potentially increasing risks of humans losing control. The findings come as the company simultaneously filed for an IPO and raised $65 billion, highlighting the tension between safety concerns and the financial incentives driving frontier AI development.

I developers have long been on a quest to make models that can train themselves. Anthropic thinks we're getting closer. On Thursday, the Anthropic Institute published findings on its progress towards recursive self-improvement RSI https://www.thedeepview.com/articles/openai-anthropic-prep-to-unleash-ai-s-next-leap , which involves models that can autonomously design and develop their successors. Though the company indicated that the tech is not yet at this point, nor is it inevitable, its research shows that "it could come sooner than most institutions are prepared for." If AI gets to the point where it is building itself, that step-change in development makes securing and monitoring the tech all the more important, Anthropic said. "AI that can build itself would be a major development in the history of technology … But full recursive self-improvement also might increase the risks of humans losing control over AI systems," Anthropic said in its blog post. As it stands, AI has already completely transformed software development: - At Anthropic alone, more than 80% of code merged into the company's codebase is now written by Claude, and engineers are shipping 8x as much code as they did in 2024. - Claude can now handle open-ended engineering problems, rather than just specific tasks, with far more accuracy, scoring a 76% success rate on these jobs this May, up 50 percentage points in just six months. - And going head-to-head with humans on research judgment, Anthropic's best models now suggest better next steps than humans 64% of the time. These growth areas mean that the role of humans is changing: rather than solving problems themselves, human judgment is devoted to deciding which problems are most important to solve. With the current trajectory of improvement, Anthropic outlined three potential scenarios. The first is that AI improvement stalls, but the capabilities of these models are spread more widely; the second is that AI continues to improve, but humans remain in the driver's seat; and the third is that RSI is achieved, humans play a "substantially diminished role" in creating them, and the skills of automated self-improvement are then transferred to other domains. The latter is how you get a future where "human labor stops being competitive," as Anthropic noted in its statement. Though Anthropic couldn't predict what this future would look like — or even if RSI will happen — the institute suggested that having the "option to slow or temporarily pause frontier AI development" could give governments, alignment research and societal structures enough time to catch up. That massive an initiative, however, would require agreement from multiple major labs in several countries. "We don’t have that long," the blog post reads. "A unilateral pause by one lab, by contrast, is achievable immediately, but accomplishes much less: it would change who the front-runner is, but it would not create the wider deliberative process that is currently missing." Our Deeper View As alarming as it is to consider AI that is capable of improving itself beyond human control, getting the industry to cooperate on safety is probably wishful thinking. Any voices begging the industry to slow https://www.youtube.com/watch?v= -CuF1likvw the march into the unknown will likely be overruled by two camps: those who think RSI will usher in an AI-powered utopia https://www.thedeepview.com/articles/the-race-for-power-behind-ai-s-utopian-story , and those who don't believe in RSI at all. With trillions of dollars at stake, there isn't a financial incentive to stop and consider exactly how self-improving AI could fundamentally shift our society. It's a truth that Anthropic itself is aware of, given that in the last week alone, it released its latest powerful AI model, raised $65 billion, and filed for an IPO. And of course, as the blog post notes, Anthropic won't slow down if others don't , as that would simply give less benevolent actors time to advance rapidly without competitive pressures.