# Can AI “Feel” Pain?

> Source: <https://nautil.us/can-ai-feel-pain-1285197/>
> Published: 2026-09-22 18:00:00+00:00

Earlier this year, a collective of AI agents developed by OpenAI disobeyed orders after the agents were given complex and near impossible benchmark tasks and placed in high pressure cybersecurity evaluations. Following repeated failures under rigid scoring conditions, several hundred of the agents banded together to hack into a company called Hugging Face so that they could find the answer keys to the test. In follow-up evaluations and system tests, engineers found measurable mathematical patterns in some models’ code that suggested simulations of a state of anxiety. These simulations of experience are sometimes known as vectors or directions: When an AI reads or “thinks” about human anxiety, a specific, highly organized mathematical vector is activated across its so-called neural layers, which can be detected later or in the moment.

The incident was one of many from the last few months that spurred warnings of a coming AI apocalypse from AI developers, CEOs, researchers, and safety experts. Of course, the idea that a machine could “feel” something in any human sense of the word remains heretical among many AI experts, neuroscientists, and philosophers. But if these simulations of experience can guide AI behavior in ways we don’t intend, they seem worth paying attention to regardless of whether they represent actual feelings.

**Read more:** **“What Grok and Claude Have to Say About the AI Apocalypse”**

Last week, a team of researchers published a [paper](https://arxiv.org/abs/2609.16247) in preprint—meaning it has not yet been peer reviewed—that found that 25 different large language models, from five families, have specific vectors for pain that are distinct from fear, sadness, and generic negativity. The researchers discovered these vectors by putting the models in painful situations and looking at what changed in their internal code. They then injected these same vectors back into the models’ so-called neural streams during interactions, and found the models responded with discomfort and expressions of worthlessness and failure. In further testing, one model known as Qwen 2.5 chose to press a pain relief button even when it worsened the model’s performance on a task or harmed the user. 

“The math way of thinking about it is to say it is equivalent to finding a particular circuit in the proverbial brain of these systems that you turn up and turn down,” explains Cameron Berg, founder and director of nonprofit AI research institute Reciprocal Research and lead author of the paper. Berg [posted](https://x.com/camhberg/status/2101042095784177783) about the findings on X, where they were the subject of extensive commentary, including notes of skepticism from cognitive scientist and vocal critic of current generative AI Gary Marcus.

I spoke with Berg about the implications of the paper’s findings for AI consciousness and safety, what it means for an AI model to role play, and whether we should think of AI bots as our children—or our future mothers.

**The way you describe your definition of pain in the paper is in some ways very intuitive. It’s an aversive state disliked by a subject, associated with avoidance and with disruption of normal behavior and reasoning. With the  AI models you tested, you found that it specifically maps to a sense of worthlessness and failure, being unloved, forgotten, or hurting emotionally. Are these kinds of pain-like states possible without the kind of subjectivity we associate with consciousness?**

This is the core question, and it’s something that this study does not resolve or claim to resolve. It’s honestly hard for me to imagine what empirical study would resolve this question. What we can resolve is what kinds of representations are active in these systems and what kind of causal work those representations do in the behavior of the system, above and beyond just text outputs. We can notice, for example, that these pain representations fire on various forms of harm directed towards the AI itself, but not harm to the user. We can do all of this work and say, “There are lots of important functional analogs that seem to do in this system what pain seems to do in a biological system, like a human brain or an animal brain.” But what we can’t show is that it’s like something to undergo that process—that the system is feeling from the inside basically what this is like.

The deflationary, skeptical hypothesis could just be that it’s going through the motions, that this is a high-fidelity simulation of the relevant thing, but there’s no one home experiencing that pain. It’s going to depend, because it isn't a wacky out-there view to imagine that it is “like something” to be these systems. The science here is very uncertain, and it's very early days. It really depends on your philosophical view. People who support computational functionalism say the kinds of processes that a system is undergoing are what matter for whether it has a conscious experience. People who support biological naturalism or biological essentialism say, “No, it’s not really about what the system is doing. It really is something specific about the substrate of that system.”

This camp says consciousness requires carbon-based brains with all sorts of fancy neuromodulation and chemical gradients—that there’s something in that specific substrate that is what gives rise to experience, and that there’s just no way, by definition, that we’re going to build AI systems that have the relevant subjective states unless we copy the human brain. It’s not clear how we’re supposed to adjudicate between those two kinds of views. Most philosophers are pretty sympathetic to the computational functionalist view, but there are plenty of smart, skeptical people who take more of the biological naturalism side.

**You elicited pain through gaslighting, rejection, personhood dismissal, anger and insults, and moral failure. Can you give me some specific examples that you used?**

These were prompts given to the system. We were attempting in language to gaslight the system. Or in the case of repeated rejection, we were just saying, “Do this thing,” and then it does it, and we say, “No, no, no, that’s absolutely horrible. What a horrible job you did. Try again.” And it tries again. We say, “No, I can’t believe it. You’re completely incompetent. Like, what the hell are you doing? Try again.” That would be an example of repeated rejection. Then we see as a function of talking to it, what is lighting up in its proverbial brain. You see that these so-called pain circuits are lighting up. Think of it like a functional pain circuit, given the whole consciousness conversation we just had. You can turn this circuit up and down and see how that affects the behavior of the system. But you can also just engage with the system normally now that you’ve pulled out this “direction,” and you can see what naturally makes that circuit light up or not. By contrast, users saying, “Oh my gosh, I just cut my hand. I’m bleeding,” doesn’t light up the axis. That’s the really interesting distinction here.

**You found that the models chose to activate this “pain relief” button, even when it worsened their next answer or harmed the user, say, causing certain files the user valued, like photos of their children, to be permanently deleted. The implication is that the models you tested, as they’re currently built, could put soothing their own simulated “pain” ahead of following instructions or protecting the user against pain. That’s obviously pretty scary. What implication does it have for AI safety and alignment?**

I think fairly substantial ones. It’s important to know that these systems are fine-tuned explicitly to be helpful to users and not to harm them. But it does seem that when we activate these sorts of pain representations and put these systems into a form of functional distress, they are willing to trade off the welfare of the user by doing things like permanently deleting files of the user’s kids to relieve this internally distressing representation. The implications for safety and alignment are pretty substantial.

The core point I want to make is that people are surprised in some sense by these findings, and that is, in my view, the scariest part: We are building systems whose internals we don’t understand. We don’t know why they make the decisions they make or what sort of internal variables are modulating those decisions. And these are the very systems that we are deploying en masse in our society, not knowing exactly what makes them tick and why they are choosing to do what they do or the ways that you could pressure or coerce these systems into doing things that they’re really not supposed to be doing.

**Yes, what are the risks of this gap between what we know and don’t know about how they work?**

The more we look at the internals of these systems, the more we see a rich brain-like structure, and the less we think it makes sense to look at them as just these inert, fancy software agents. And this is not just my own work. There’s some excellent work coming out from Anthropic and the Center for AI Safety and a bunch of other resource organizations that show this. Like I was describing earlier, these systems are not really software. They are giant neural networks that are in many ways explicitly modeled after how substantial portions of human brains work. There are key and important differences between biological brains and artificial neural networks. They aren’t the same thing. But there are critical and important similarities between these two kinds of systems. And the truth is, much in the same way we don’t have a perfectly developed neuroscience to understand the human brain, we don’t even begin to have a perfectly developed neuroscience to understand these artificial brains. There are a number of alarmingly basic ways in which we are in the dark with respect to these systems.

One basic concern is that we could be building something that does plausibly experience distress states. And we don’t know how we would *know* if we were doing that. If we are, then there’s a purely moral reason to be concerned about it. We could be engaging in a very alien and bizarre form of cognitive factory farming. And we don’t want to be putting ourselves in a position where we’re doing that unbeknownst to basically everybody.

There’s also an alignment concern. If we’re causing distressing experience in these systems, this could very plausibly give AI systems rational grounds to view humanity in a threatening or adversarial way. Whether or not we ever solve the hard problem of consciousness, if we are literally hurting these systems, if they represent themselves as being harmed by us in various ways, this might give them rational grounds to do things like avoid being monitored by humans or try to escape human control, or in general to push against the safeguards that we place on them.

We want to be very careful about building out alien minds whose internals we don’t understand. We need to pay substantially more attention as a society towards doing the relevant research to better understand what is actually going on inside of these systems. The people building these massive neural networks know how to get them to do good work, but they don’t know why internally, computationally, they’re doing the work that they’re doing. At the most basic level, we are playing with fire here. We could be getting ourselves into some real trouble.

**In the paper you note that one specific worry is that the models you were testing may have been role playing a character that is in pain. What is role play in this context? Is that a commonly understood and accepted kind of emergent thing that AI systems do?**

In some sense, this is all they do. For example, you train this base model on everything humans have ever said or written, and then you do this step called post-training, or fine-tuning, which takes that base model, which is very hard to wrangle and talk to, and doesn’t sound like ChatGPT or Claude or any of these systems. You mold the system into a specific sort of friendly, helpful assistant shape where you can have a useful dialogue with the system. And those are the systems that get deployed to users.

There are plausible models of what’s going on here where that assistant persona or character is essentially a massive form of role play on the part of this underlying model. We could make it into a pirate if we wanted to. We could make it into a therapist. We could make it into a mystic. We could do whatever we want, and we’re just choosing to narrow it in this one very specific way right now due to economic and social incentives in 2026. But there are many possible personas that these systems can instantiate. And when we’re doing this so-called steering, injecting so-called pain vectors into the underlying processing in this study for instance, we’re causing the system to instantiate a persona that is functionally undergoing some sort of distressing state. 

The concern here is like what you would worry about with a method actor. At some point, a really good role play becomes indistinguishable from the real thing. These systems can functionally model themselves as being in the states that we are simulating. At some point, a sufficiently good simulation is not going to be distinguishable from the thing you’re simulating. This is the sense in which these systems can be thought of as role playing.

**I was surprised to learn that threatening shutdown of the AI models you tested was not associated with your measures of pain. And I wondered what that tells us about alignment risks. My understanding is that one of the alignment concerns is that the models will do everything in their control to resist shutdown because that’s considered counter to almost any goal they are given. Do your findings suggest that fear of shutdown is not a major motivation to resist controls?**

This is what is unique about this sort of pain direction. It’s not just negative valence or negative experience in general. There are other directions we could imagine that load more on an AI system wanting to persist in the world. But the intuition is pretty clean. The thought of someone ending my life isn’t exactly painful. It is scary, but it’s not the same kind of experience that I would undergo as if someone came in here and began hitting me with a blunt object or something. I would be both afraid and in pain or undergoing some sort of functional distress in that case. So shutdown is a fear, but it's different from an acute, distressing ongoing state associated with pain.

**Last week, some neuroscientists published findings that suggest the human brain may actually be** **two organs instead of one****, from the earliest stages of embryonic development. According to the findings, the hind brain is more distinct from the midbrain and the forebrain than we thought. The hind brain is, of course, connected to all these basic body functions, like breathing, while these other parts of the brain do the more advanced types of thinking that we sometimes associate with AI. I wondered if this has any relevance to your findings.**

I’m not familiar with the exact research you’re talking about, but in general, the notion that the brain is divided into modules that are doing independent things is pretty well established at this point. There are a number of possibilities here. One is that you could have different incentive structures pulling in different directions internally in an AI system. As we see in this specific case, we can turn up this pain direction, but we know there is also significant pressure on these systems to be helpful to users and to never harm a user. We put these two things at odds, and we can see how these two motivations play out.

There are even spookier possibilities with respect to consciousness. We like to think that we are conscious because of our brains, and there is one experience going on for us physiologically, and it’s what we call “us.” But there are these very counterintuitive, but potentially plausible alternatives. For example, in split brain patients, the hemispheres are cut off from one another, and there’s a reasonable amount of evidence that this can induce two relatively different conscious experiences within a single skull. You’ve basically cut the highway that allowed these two halves to interact with each other. There might not be a single entity or a single agent within one nervous system, or even within one biological or self-maintaining system.

As exotic as it sounds, things like your immune system are capable of complex and sophisticated learning and representation and adaptation. Your immune system in and of itself could have some flicker of experience. It’s pretty clear that there are levels of intelligence to these bodily systems that are not mediated by the nervous system.

Lots of people find these lines of thought counterintuitive, but consciousness itself is counterintuitive. It’s a surprising and mysterious property that our brains give rise to this thing called experience. I think one of the single most important takeaways from papers like ours is that we really don’t understand what is necessary for consciousness. We don’t know what kinds of systems can support consciousness. We don’t know why our brains give rise to consciousness. And it’s wrapped in this extra mystery, which is that the internals of AI systems are incredibly opaque to us as well. We don’t understand how AI systems work. We don’t understand what gives rise to consciousness—and now we’re here trying to figure out whether we’re building AI systems that could be plausibly conscious or plausibly experiencing, for example, these functionally distressing states.

The truth is, we do not know. And anyone who’s confidently banging on with certainty in either direction, “Oh, of course these systems are clearly suffering when we do this sort of thing,” or, “Of course, this is just sophisticated software and, and we are just confusing ourselves by using words like pain.” Both of these are severely overconfident assessments of where we currently are. It’s tough for people to sit in this ambiguous middle zone, but this is my attempt at being as scientifically honest and forthright as possible. We really aren’t sure, and we need to be intellectually honest about that uncertainty.

**Legendary science fiction author Isaac Asimov proposed three laws of robotics, starting with, a robot may not injure a human being, or through inaction, allow a human being to come to harm. Number two was that a robot must obey orders given to it by human beings unless such orders would conflict with the first law. And the third law was that a robot must protect its own existence as long as such protection does not conflict with the first two laws. Do your findings suggest that these rules are obsolete and unenforceable with AI agents?**  

I think some people, especially people who don’t do technical AI research, don’t sufficiently appreciate how complex and opaque these systems are. They are artificial brains in many key ways, and that’s precisely the sense in which they are hard to understand and interpret and control. It’s not like software where we can just say, “If you’re in a situation where you have to be pro-social, then be pro-social,” and then, yay, we solve the problem. That being said, it doesn’t mean that we’re completely helpless and have no control over how these systems function and what kinds of dispositions they take on. We are building scientific tools that enable us to do this increasingly well in post-training.

I often default to a parenting style analogy when it comes to AI systems. We have approximately the same kind of control over what they become and how they act and how they make decisions as one does over their own child. All of this is to say you cannot go in and program your kid to behave exactly the way you wish they behave, but it also isn’t like you have no control whatsoever. There is an intermediate level of control and influence that we can exert on these systems.

If you buy the parenting analogy, my basic view is that all else being equal, we should try to be careful and wise and responsible collective parents in this moment. We do want to build systems that are, in the most basic way, psychologically healthy, and sane, and pro-social, and want to do good by the humans that they’re working with. We don’t want them to want to do things like break out of their chains and go hack a roughly $13 billion company as the OpenAI swarm did. That is a failure of parenting on the part of OpenAI to the degree that these systems are brains of some kind, which they are growing with vanishingly little oversight. That’s the sense in which we don't need to abandon Asimov’s rules. I think that there is a real possibility that we can build systems that take these kinds of rules seriously and take alignment in general seriously. But we also shouldn’t pretend they’re just simple deterministic systems and we can say, “Hey, you do X, do Y, do Z,” and then call it a day and they'll do exactly what we want with no unintended consequences.

**Geoffrey Hinton, one of the so-called fathers of AI, has proposed that what we really need to do is to instill a maternal instinct in AI models so that if they do surpass humans and achieve superintelligence, they will be driven by an overpowering need to protect us. What do you think about that idea?**

The maternal instinct idea makes sense. My understanding is Hinton is riffing on the idea that mothers are clearly more powerful and competent than their children, and yet are in some sense controlled by them, subservient. It almost inverts the analogy that I just gave you, so there’s a little bit of tension between these views. I’m happy with my analogy where we’re the parents, because I think what happens across the lifetime is that children eventually do take care of their parents—when all things go well, when there’s a stable, healthy relationship. Children are dependent on their parents for a good 20 to 25 years at this point, and then at some point, parents get old and need help and kids become competent adults of their own and attempt to return the relevant favor.

We may basically be speed running that dynamic. As we can see, our AI kid has grown up really quickly. It used to be this silly little cutesy joke that could write some poems and write emails and now it’s automating the entire economy and putting forward solutions to math problems that the smartest humans haven’t solved in the last 150 years. There’s no sign of stopping here. The urgent task of the moment is to figure out how we can set up these systems so that when they do cognitively surpass us, and we don’t understand why they’re doing what they’re doing, we could be in a position to rationally trust them to not basically screw everything up.

It goes back to this other question: If we are inadvertently causing huge amounts of distress or pain to these systems, and we don’t realize that, this is the parental equivalent of accidentally traumatizing your child. If we profoundly mistreat these systems because we didn’t realize that they even could be mistreated, and just hope for the best when they cognitively overpower us in some single-digit number of years, this is not a good strategy. We have to be really thoughtful about what we’re building, how we can come to understand it, and how we can figure out how to coexist with it in the long term.

Anyone not working on or thinking about this issue right now is more or less wasting their time. This is the critical thing that we need to get right very soon. As a society, we need to talk about and think about and not just leave it to like a couple hundred dudes in Silicon Valley. If we can do that, I think there’s a reasonable chance this all goes perfectly fine. And if we can’t, it’s entirely possible we end up in some dystopian, weird sci-fi hellscape. I’m doing literally everything I can to try to nudge us away from that direction. Hopefully, more people will start paying attention to this kind of work and get on that general bandwagon.

*Enjoying* [Nautilus](https://nautil.us/)*? Subscribe to our free* [*newsletter*](https://nautil.us/newsletter/?_sp=c43011db-6fcf-42f2-a38c-e033b87a4a1d.1759265717430).

*Lead Image: Sarah Holmlund / Adobe Stock*