- See more Daily Mail on Google - save us as a Preferred Source A researcher at Anthropic has dramatically quit, following concerns over the 'out of control' superintelligence race.
Jacob Coxon has spent the last three years training new AI models, both at OpenAI and Anthropic.
However, he now claims that neither company is 'acting responsibly', and even warns that AI could 'kill all humans' by the end of the decade.
Taking to X, he explained: 'I resigned from Anthropic today.
'I spent the last three years doing pretraining research at both OpenAI and Anthropic.
'Neither company is acting responsibly. They are racing straight to self–improving superintelligence and gambling with our lives.'
A researcher at Anthropic has dramatically quit, following concerns over the 'out of control' superintelligence race
In a series of tweets explaining his decision to leave Anthropic, Mr Coxon urged people to 'not underestimate the power' of AI – particularly if it becomes 'superintelligent'
In a series of tweets explaining his decision to leave Anthropic, Mr Coxon urged people to 'not underestimate the power' of AI – particularly if it becomes 'superintelligent'.
Superintelligence is the point at which an artificial system becomes more powerful than any individual, company or even nation.
'These will soon be superhuman systems that can hack anything, revolutionize any field overnight, and acquire real power and resources,' Mr Coxon said.
'We have all witnessed the progress in each of these domains, and progress is not slowing.'
While the idea of AI killing humans might sound farfetched, Mr Coxon claims it could become a reality in just a few years. 'The people building AI earnestly believe that it could kill us all by the end of the decade,' he said.
'This is not a marketing stunt. If anything, many executives and senior researchers will couch their phrasing in the press to sound sensible – but I hear the same people express fear privately.
'No other human activity poses this level of danger.'
In response to Mr Coxon, Evan Hubinger, Alignment Science lead at Anthropic, confirmed that the firm believes AI has the potential to kill humans
According to Mr Coxon, this danger is 'well–understood' at Anthropic.
However, the company is 'locked in a race to get there first', the researcher adds.
He explained: 'Accepting this race and entering the "endgame" is a hubristic gamble that should not be launched from a private company's Slack.
'Attempting to speedrun alignment should require extraordinary confidence that there are no better trajectories available.'
As an example, the researcher highlights the recent Hugging Face attack, which saw a firm hacked by OpenAI's rogue AI.
He said: 'Warning shots like the Hugging Face attack have made pacing agreements between U.S. labs more viable.
'I don't feel like we're on track to prevent a global race, which may require costly actions such as a temporary ban on improving model capabilities.'
To conclude, he urged fellow AI researchers to 'consider what the next few years will actually look like.'
While the idea of AI killing humans might sound farfetched, Mr Coxon claims it could become a reality in just a few years. Pictured: Terminator Genisys He asked: 'Do you want to kick off a superintelligent RL run without a rigorous understanding of its mind? Should you put your head down because "it's happening anyway" – or take this moment to call for different conditions?'
In response to Mr Coxon, Evan Hubinger, Alignment Science lead at Anthropic, confirmed that the firm believes AI has the potential to kill humans.
On X, he said: 'Jacob is correct here – we really do earnestly believe AI could kill all humans! I personally think it is >10% within the next decade. I believe Anthropic is trying its best, but we do not yet have a plan to solve alignment for superintelligence and are not clearly on track to.'
Mr Coxon's warning comes just days after Geoffrey Hinton – a Canadian researcher often referred to as the 'Godfather of AI' – warned that superintelligent systems could 'lead to human extinction'.
'We would be very foolish to develop superintelligence now, when there is no scientific consensus it can be developed safely and controllably,' Dr Hinton said.
'Losing control over AI smarter than ourselves could be catastrophic and could even lead to human extinction.'