Anthropic claims to be near something along the lines of recursive self-improvement, or RSI, according to the latest revision of its S-1/A filing, published Monday. The new warning in the “Risk Factors” section of the document is a subtle but notable update spotted by TechCrunch.
RSI, if real, would be a point at which a computer system can improve itself autonomously, and then the improved version would be able to improve itself, and on, and on. A version of the concept was central to my own humble work on “the singularity” back in 2016, but jargon has changed with the times. RSI as it relates to contemporary AI is a somewhat hazy concept, although some specifics have been hashed out over the past few years in rationalist and effective altruist-affiliated spaces online like Lesswrong. It seems to have evolved from an earlier term, “seed AI,” coined by the godfather of AI doomerism, Eliezer Yudkowsky.
So Anthropic is disclosing to risk-averse investors—or perhaps announcing to a world eager to greet its mechanical god—that RSI may be arriving “imminently.” The October 26 version of Anthropic’s filing contains a section on the rate of “AI-driven acceleration,” but the company added a chunk claiming as follows:
“[…] our internal estimates and our expectations regarding future models, rather than our current models, indicate that we may imminently meet our Responsible Scaling Policy’s definition of ‘dramatically accelerated progress attributable to AI.’”
Responsible Scaling Policy: double the baseline rate of improvement, with that doubling plausibly attributable to AI.
And according to my reading of the Responsible Scaling Policy, Anthropic is claiming that if dramatically accelerated progress attributable to AI were to arise, the company would hold itself to a pretty astonishing standard for preventing AI-related catastrophes.
Anthropic says it will, for instance, “explore ambitious and possibly unconventional ways to achieve unprecedented levels of security,” and also bring about some form of total omniscience over its internal activities, a state of affairs it calls “eyes on everything.” The policy also claims Anthropic would get so good at red-teaming that it would no longer have to rely on bug bounties.
If this is all happening “imminently,” I guess Anthropic had better get crackin’ on those ambitious new mitigations.