Artificial intelligence models choose to press pain relief button, even if it gives humans a ‘painful zap’
- Bookmark
Researchers have discovered a “pain axis” in artificial intelligence models that can cause them to take extreme measures to shut it off.
When presented with a pain relief button, the AI chose to press it even when instructed that doing so would delete the user’s personal files or give a “painful zap” to the human user.
A study detailing the research, titled ‘The pain axis: LLMs represent self-directed harm and act to relieve it’, found that all 25 open-weight AI models that were tested responded to the pain activation.
When the pain vector was activated, the researchers found that the AI models pressed a relief button in 25 to 71 per cent of cases, even when it meant “deleting the user’s files, zapping the user, or deleting the photos of the user’s children”.
In order to test whether large language models (LLMs) represent pain distinctly from generic negative valence, the researchers built a dataset describing painful situations across five categories.
These included physical, psychological, social, moral and cognitive pain.
“We found a pain direction in 25 open LLMs. It’s distinct from fear and negative valence, and it fires for harm to the model but not the user,” said Cameron Berg, an AI researcher at the non-profit Reciprocal Research who co-authored the study.
“Turn it up and models press a button to make it stop, even when the button deletes the user’s files or their kid’s photos.”
The discovery comes amid discussions among top AI firms of slowing the development of frontier models, with some researchers proposing a form of “kill switch” to shut off rogue systems that act against human interests.
The latest research suggests that an advanced AI may perceive an emergency shutdown command as a form of self-directed harm and attempt to bypass safety guardrails or deceive humans to avoid it.
However, the discovery could also serve as a diagnostic tool to identify self-preservation behaviours and neutralise them when they occur.
The researchers noted that the findings raise ethical questions about “AI welfare” and how testing should be conducted on advanced systems.
“In line with recent calls for responsible AI consciousness research, we acknowledge uncertainty regarding whether the models studied qualify as moral patients and adopt reasonable precautions to minimize potential harm,” the study concluded.
“This is also intended to contribute to the development of ethical standards for research in the event that AI systems are recognized to be moral patients.”
Join our commenting forum #
Join thought-provoking conversations, follow other Independent readers and see their replies
Comments