Where does the race to automate AI research end?

A recent MATS research talk argued that the imminent automation of AI research, as predicted by OpenAI and Anthropic, could cause an unrecoverable alignment failure. The talk identified three dangerous properties: oversight breakdown at scale, self-amplifying capabilities, and asymmetric acceleration of capabilities over alignment. The outcome, according to the researcher, could be lethal and irreversible.

This is a linkpost of a recording of a recent MATS research talk where I argue that the automation of AI research — which OpenAI and Anthropic say is imminent — could lead to an unrecoverable alignment failure. Three properties make it especially dangerous: oversight breaks down at scale, capabilities self-amplify, and capabilities will be sped up asymmetrically faster than alignment. The outcome could be a lethal, unrecoverable alignment failure. Link to the paper preprint https://storage.googleapis.com/papers-preprint-simonlermen/rsi paper preprint.pdf .