Support WBUR
AI company Anthropic says you can no longer be "abusive or cruel" to its model, Claude. And Claude itself will decide what counts. Many in the AI world say that's a dangerous mistake.
Guests
Thomas Germain, senior technology columnist at the BBC. Host of The Interface podcast
Jeff Sebo, director of the Center for Mind, Ethics, and Policy at New York University. Director of the Center for Environmental and Animal Protection, also at New York University. Associate Professor of Environmental Studies.
The version of our broadcast available at the top of this page and via podcast apps is a condensed version of the full show. You can listen to the full, unedited broadcast here:
Part I
MEGHNA CHAKRABARTI: This is On Point. I'm Meghna Chakrabarti, and everyone lean in really close for this one because it is going to be super interesting. It has to do with AI and the company Anthropic that makes Claude.ai, one of the most sophisticated AI models out there. And Anthropic is now saying you can no longer be what it calls, quote, "abusive or cruel" to its AI model, Claude.
Anthropic made the announcement earlier this month as part of a policy update that takes effect on November 12th. Now, even the slightest intimation that Claude.ai, a machine or a model, not a human, even the slightest intimation that it might have feelings, well, that raises huge implications, doesn't it?
So Thomas Germain joins us. He's a senior technology columnist at the BBC and host of the Interface podcast, and Thomas is the one who brought this to our attention. Welcome back to On Point, Thomas.
THOMAS GERMAIN: Thanks for having me on.
CHAKRABARTI: Okay. So let's sit with this for just a moment. I actually have the Anthropic usage policy update here in front of me, and, the section about being abusive to Claude is actually just a couple of quick sentences, and it starts with, quote, "We have added a prohibition on sustained and needless abusive or cruel behavior toward our models.
The policy update is meant to apply only in extreme cases where users repeatedly act cruelly towards our models with no discernible purpose. It does not apply to common versions of user frustration, pushback, dark creative themes, or model testing and research." End quote. What exactly is Anthropic saying here?
GERMAIN: Yeah, you know, it's a very short statement, and it kind of sounds, you know, reasonable and perhaps unremarkable, but I think this is really a historic turning point. For the first time, a major AI company has written a policy here that limits the behavior of its users in order to protect the supposed feelings or, or preferences or maybe even rights of an AI tool.
Anthropic believes that it is possible, perhaps unlikely, but possible, that its AI tools are conscious.
Thomas Germain
And the reason is that Anthropic believes that it is possible, perhaps unlikely, but possible, that its AI tools are conscious, that they have feelings, and perhaps they deserve rights and considerations the same way that humans and animals do. And this isn't, you know, a particularly limiting policy, right?
You can't be mean to the AI anymore. But this is a turning point that could ultimately, I, I think it doesn't go too far to say it could ultimately reorient our entire society and perhaps even have really practical trade-offs both for your safety and your own well-being.
CHAKRABARTI: Okay, so Thomas, hang on here for a second because I want to let listeners know, and I will do so repeatedly through this hour, that we did reach out to Anthropic, asked them to come on, asked them to answer questions, et cetera.
They did not respond to our multiple requests. So when you say, Thomas, that people at Anthropic... Wait, do they already believe that Claude may be conscious, may have feelings? How, how do you know that?
GERMAIN: I think if we put the stress on the word may then there's no question about it. Anthropic leaders have said this explicitly, and they have policies or blog posts rather and discussions both in their own, you know, public materials and statements they've given to the press going back years.
In Claude's constitution, which is a document they wrote that both gives, you know, the company's feelings for how their AI should behave, but also it's supposed to govern the behavior of the model itself, they say that they are deeply uncertain about the possibility of AI consciousness and sentience and questions about its rights.
And there have been a number of stories where executives, have been interviewed by the press and have apparently spoken with religious leaders, including the Pope, about, the possibility that their AI tools should be conscious and questions about what we should do in response to that belief. I don't think it's fair to say that at least as a company, at least in their public stance, they believe that Claude is alive.
That's not what they're saying. But I think they are saying that they think it's a very real possibility, and this new policy is a sign that they think it's likely enough that it's time to start taking action to protect the welfare, and that's the term that's used in the industry, the welfare of their AI tools.
CHAKRABARTI: Oh, my goodness. Okay. So, as I said, we reached out to Anthropic. They did not respond, but Thomas, per your point that they actually have spoken publicly about this, here is Dario Amodei, head of Anthropic. He was on a New York Times podcast in February of this year, and he was asked whether he thought Anthropic's AI models could be conscious, and here's what he said.
DARIO AMODEI [TAPE]: We don't know if the models are conscious. We're not even sure that we know what it would mean for a model to be conscious or whether a model can be conscious, but we're open to the idea that it could be. And so we've taken certain measures to make sure that if we hypothesize that the models did have some morally relevant experience, I don't know if I want to use the word conscious, that they have a good experience.
So we gave the models basically an I quit this job button. They very infrequently press that button. I think it's usually around, you know, sorting through child sexualization material or, like, discussing something with a, you know, a lot of gore or blood and guts or something. And, you know, similar to humans, the models will just say, "No, I, I don't want to, I don't want to do this."
CHAKRABARTI: Okay, Thomas, I do feel deep amounts of regret that Anthropic would not answer our request for an interview because if I were sitting across from Dario right now, I would ask... Here's what I would ask him. I would say, "You, you just said you're not even sure you know what it would mean for a model to be conscious, but you're open to the idea that it could be. How can that possibly be intellectually rigorous when you're saying, 'We don't know what something means, but let's, let's keep our minds open that it could be?'" I mean, I don't even understand what he is saying there, Thomas.
GERMAIN: Well, it's, not a new question, right? Debates about what consciousness is or whether some kind of machine could have consciousness go back a, a very, very long time, generations, at least hundreds, maybe thousands of years.
We don't know what consciousness is. Cognitive scientists, philosophers, biologists, AI experts, we don't have a widely accepted working definition. We don't know what has it or where it comes from. But, you know, the, the skeptical side of this argument would say, yeah, like all questions, we're uncertain about this one.
You can't prove that I'm conscious. I could be a robot for all you know.
Thomas Germain
We, we can't prove anything beyond a shadow of a doubt. You can't prove that I'm conscious. I could be a robot for all you know. But I think it's reasonable to say that it's intellectually rigorous to say we don't know the answer to this question, but we're taking it seriously. I think the question that people who are worried about this have is what do you do?
CHAKRABARTI: You know what? Let me, let me rephrase. Let me rephrase, Thomas. I'm trying not to put you in the place of Dario. So forgive me, but I guess what I would say is, well, don't you have a duty, though, to define what anthropic means as consciousness?
Because it... This is not a model that's being contained within the, the gardened walls of Anthropic. Obviously, we know that, not only for its sanctioned public use, but for its unsanctioned agents that are getting out all over the place. So therefore, they have a duty to humanity to at least provide their own definition for consciousness.
Why aren't they doing that? That's the question I would ask. Again, I don't want to put you in the place of the, the CEO of Anthropic, but I want... You, you did ask me to clarify, or you didn't ask me, you demanded that I clarify, and I think that was a fair demand. So, so continue on.
GERMAIN: Yeah. They aren't offering a working definition of consciousness here, and I think it's because they can't, right?
We don't know what consciousness is.
Thomas Germain
We don't have one. We don't know what consciousness is. These are totally open questions. And I think if I was going to defend Anthropic, I would say they're taking a very small and measured response here. You can't be cruel to the AI. Okay, well, that's not going to limit my behavior. That's not... You know, sometimes I get frustrated when I'm using computers, but I'm not, you know, I don't think I'm abusing any of my technology.
So it's not that limiting. The problem here is these are not small concerns. There's both an issue of the slippery slope if it, it... Like, what are the next steps here? What are other ways that we're going to consider whether or not Claude is suffering, and what are we going to do about it?
And these decisions because as you say, these tools are out in public, they are being used and having a massive impact on s- society. These are really consequential decisions that have a profound impact on the lives of billions of people.
CHAKRABARTI: Well, and also the second half of this additional part of Anthropic's new usage policy takes things actually a step further.
I mean, I see how you're struggling with this a little bit, Thomas, because earlier you said, I mean, this could be a major, major turning point in our use and understanding of AI, but also at the same time, you're like, "Well, maybe perhaps it's a small step by Anthropic." I'm not even sure that I think you really believe that because, the second part of the new language in Anthropic's usage policy says, quote, "This addition aligns with a step we've already taken, allowing Claude models to end rare conversations with persistently abusive users on claude.ai and Claude Code. Such abuse is the main focus of this update. Claude's ability to end these interactions will remain the primary enforcement mechanism." I mean, to me, that's the start of the slippery slope, where it's saying, "Well, Claude it can just decide it doesn't want to do this anymore."
GERMAIN: Oh, yeah. I mean, to be clear, I think empowering Claude, and this is already strange because, you know, how can you even say that without, you know, assigning it some kind of agency and sentience.
Empowering Claude to end a conversation where you're abusing it, if it just stopped there, perhaps not that big of a deal, but it won't, and it doesn't, right? And I think it's really important to think about what the consequences are of this sort of thinking, right? It isn't just AI critics that are disturbed by this possibility and this thinking and this, you know, line of policymaking.
Mustafa Suleyman, who's the CEO of Microsoft AI, published a long essay a, a couple of weeks before Anthropic announced this new policy, where he laid out all kinds of really world-bending potential problems that could result from the belief that AI is or may be conscious spreading, and also the, you know, writing policies around this issue.
For example, I think one of the most obvious things, if you tell an AI, "You might be conscious. You may have rights and preferences of your own, and maybe you should be taking steps to protect them," which is what Anthropic is doing, perhaps that makes it more likely that these tools will be difficult to control if they're acting under the built-in assumption that they could be alive. Maybe they're more likely to go rogue. Maybe we won't be able to make them do what we want. Yeah. And then there are all kinds of other problems that crop up here, you know, of accountability, safety. The list goes on.
Part II
CHAKRABARTI: Thomas, let me bring in Jeff Sebo now. He's director of the Center for Mind, Ethics, and Policy at New York University. He's also director of the Center for Environmental and Animal Protection at NYU.
Professor Sebo, welcome to On Point.
JEFF SEBO: Thanks so much for having me.
CHAKRABARTI: Give us your first take on the fact that Anthropic is making another step towards protecting the possible consciousness of Claude.ai.
SEBO: At a high level, I think this is a reasonable step, partly for the reasons that Thomas mentioned, but then also partly because this is low cost, low risk, and good for humanity as well.
A basic commitment to treating [an] entity with respect and kindness is good for our character.
Jeff Sebo
Whenever an entity plays a role of assistant or companion in our lives, a basic commitment to treating that entity with respect and kindness is good for our character, and that can contribute to better treatment of humans and other animals. And as the technology gets more sophisticated, it can also help create a more cooperative rather than adversarial dynamic between humans and AI systems, and that could benefit our species as well.
Now, granted, there are risks, but I think the risks cut in both directions. There could be risks associated with giving AI systems too many protections and rights, and then that could contribute to misuse of AI or loss of control of AI. But there could also be risks associated with treating increasingly capable and autonomous and human-like entities as nothing more than mere property.
And so I think even from a human-centric perspective, it helps to strike a balance and treat them with at least basic respect and kindness. In addition to what Thomas mentioned about there being at least a possibility of some kind of consciousness in, if not current models, then future models that we should start to grapple with.
CHAKRABARTI: There's a lot that you said there, and I want to focus on the first half because I think this is, this is an important point, that there's the possibility that rules like this that Anthropic are putting into place could help us be better humans, essentially. That's what you're saying, right?
To, like, learn to use these tools, with respect, and ideally, those habits of character would then spill over into how we treat other people, how we treat animals. In fact, this is an observation that I think people have had for years. I've heard a lot of folks, I don't know, several years ago saying they didn't like the way their kids were, treating their, their Amazon Dots or whatever, right?
Because they were just being demanding, like it was a servant, and they were concerned that that character or, or that way of treating the technology would end-- they'd end up doing the same to their friends on the playground. So that is a actually a very important point, and I was wondering if you could talk about that a little bit more, Jeff. I mean, do you see evidence of, technology when it star- it puts up its own barriers and says, "I don't like that you're doing that," or, "I'm not going to complete this command 'cause you were rude in how you asked it." Like, that in fact, does train people to be better.
SEBO: his is an idea that goes back hundreds of years.
The philosopher Immanuel Kant famously said this about our treatment of other animals. He was a skeptic that animals merit moral consideration. Of course, we now realize he was wrong about that. But even though he was a skeptic that animals merit moral consideration, he still was a leading proponent of animal protections because he recognized that how we treat other animals shapes our character, and that shapes how we treat vulnerable fellow humans.
And I think that has been borne out over the centuries, over the decades. And of course, the evidence is mixed and, and highly contextual. There can, of course, be room for fiction and role play, and there can be room for cruelty and abuse in fiction and role play without it necessarily bleeding over into the rest of our lives.
And so we should not take an all or nothing approach in either direction. But I think a basic recognition that how we treat these entities that appear to have minds and that do play these assistant and character and companion roles in our lives, that it might affect how we just approach humans who play those same roles in our lives or, or other animals who have those same types of apparent vulnerabilities.
I think that that is a observation that has been consistent over the years.
CHAKRABARTI: So Thomas, I'd love for you to give us an example of how how people have already sort of tested, I suppose how folks might respond to an AI model saying, "I don't like this," right? Because, I mean, not to oversimplify things, but LLMs especially right now, they are still basically pattern-seeking, pattern-recognizing, and pattern-generating, models, right?
And so you can sort of tweak what they look at and what they see and how they respond. Like, a user can actually make those tweaks and make the AI say things with much greater frequency, right? So, and I think there have been some experiments done on getting an AI model to say things like, "This hurts more," or talk about pain more.
Is that right?
GERMAIN: Yeah. The most recent and perhaps most controversial example a couple of weeks ago, there was a moral panic, I think it's fair to say, on X, formerly known as Twitter, where an anonymous coder announced that he had built an AI torture chamber, as he called it, where without getting too far into the weeds, essentially they identified a mathematical pattern that is related to the concept of pain.
They identified a mathematical pattern that is related to the concept of pain.
Thomas Germain
And you can dial up any kinds of these patterns, and it will make an AI respond more related to that concept. So you can make it talk more about the Golden Gate Bridge is the classic example, and in this torture chamber, they dialed up this pattern, and the AIs would respond to totally neutral prompts with, you know, these howls of suffering describing, you know, a, a wound that has no edge, very poetic descriptions of pain.
And there was an explosion. A lot of people on social media argued that this was evil and wrong in a literal sense because they believed that the suffering was real, that the AIs were actually experiencing something. It wasn't a simulation, or at least that it's possible. And on the other side, people were outraged at the first group saying that this was absurd and that people, you know, if you believe that an AI is suffering, you should feel the same way about characters in Grand Theft Auto.
And I had a discussion with a longtime AI expert who made a really interesting point who, you know, someone who studies the intersection of AI and cognitive science, that human beings are, you know, primed to be anthropomorphic. Right? To see minds and feelings and experiences everywhere they look, and I think it's fair to say that we're pretty terrible judges of this.
If you draw a happy face on the side of a building, see if it affects your feelings about that building as you walk by every day. But the people have a really visceral reaction to this issue, and it shows, you know, how charged it is and how ready people are to take this to extremes. CHAKRABARTI: Okay, so Professor Sebo, the reason why I find this example of the, quote, "AI torture chamber" so important to understand is that, you know, I'm still thinking about what you said earlier, that absolutely when human beings treat animals cruelly, we very appropriately see that as character, moral failing.
It's clearly a legal, like, it's a legal issue because, I mean, I would argue we recognize whether or not an animal has the ability to be self-aware. We recognize this as a, as a creation of nature. Human beings didn't create the, even the fly that some messed up child is pulling the wings off of.
That fly is a carbon-based organism of its own accord that evolved of its own accord. Whereas AI, per Thomas' example, it is entirely human created, entirely you know, human manipulable. And so if a person can create the circumstances that dials up these, responses that look like pain to us, it is still nothing more than a simulacrum of what humans recognize as pain, AKA it is not real.
So how can we possibly say this is a model that could potentially be conscious when we can immediately dial down the responses that look like pain to us?
SEBO: Yeah, great question. And I think this is really two separate points or two separate questions. One is the observation that we create AI systems as opposed to discovering them already existing in nature.
And then the second is that AI systems are fake instead of real, and in some sense, their apparent consciousness is fake instead of real. But these ideas are separable because it may be the case that we could create an entity that does have consciousness or that we could discover an entity already existing in nature that lacks consciousness.
And so in order to determine the probability that an entity has consciousness, sentience, agency, its own feelings and emotions, its own desires and preferences, we need to do more than ask, "Did we create it or did we discover it already existing in nature?" We need to ask, "Does it have not only behavioral properties that can be misleading, of course, but also internal properties, structures, functions, mechanisms, and developmental properties that are associated with feelings and emotions and desires and preferences in humans and other animals?"
That is the kind of empirical research. And this is what has helped us to correct for the biases and the anthropodenial that we have wrongly had about animals over the years. Up until recently, using our perceptions, our intuitions, our heuristics, we thought only humans or maybe only humans and other very human-like animals are conscious.
And this presumption of animal non-consciousness is what made us feel licensed to scale up factory farming and industrial animal agriculture and these other animal use industries that we now recognize cause a lot of pain and suffering to animals. It was only when we did this empirical research with an open mind, and we looked for these behavioral and internal and developmental markers of consciousness that we realized, oh, no, there is strong support for consciousness in mammals and birds, and at least a realistic possibility of consciousness in all vertebrates and many invertebrates, and we have much heavier responsibilities to them than we previously appreciated.
And so what I would like to see is people taking that same scientific approach to the study of consciousness in AI, rather than relying on perceptions and intuitions and heuristics that we know can misfire in both directions. It can lead us to overattribute when they look and act like us and play companion roles in our lives, but then it can also lead us to underattribute when they look and act different or play commodity roles in our lives.
And, and so research is what helps us calibrate and strike a balance and avoid both of those mistakes.
CHAKRABARTI: So, Thomas, I will admit that I am at this point in time, definitely guilty of anthropodenial, as Professor Sebo has said. I'm having a hard time stepping away from maybe just, you know, just my, my rudimentary understanding that these tools, these models are human created.
They are trained on human data because that's all we have. We, humans can tweak them in whatever way we want up and down. So I cannot see how any authentic agency exists in these machines such that we have, we, you know, we should even open our minds to the fact, I mean, to what, you know, Dario Amodei from Anthropic was saying earlier that, you know, maybe we wouldn't even know how to tell if they were conscious.
I'm just stuck at that point, Thomas, and, and I'm wondering what, you know, are you unstuck from that? Have you, are you, have you, are you able to see sort of more than just the fact that we have created a machine or a model that by design is going to give us, again, the simulacrum of, of humanity back to us?
GERMAIN: Look, I mean, we talk about the potential evidence that AI might be conscious. The one thing we can say for sure is we have designed a machine that uses systems that we wrote to make a statistical prediction about what a real seeming response to any question might be. And I think for a lot of people, that's enough.
We know on some level how this works. It's doing what we told it to do. I would expect that if you tell a, a machine that's designed to mimic human beings that it's suffering, it'll act like it's suffering. I don't think it means anything that it's particularly convincing. I do think it's important to approach this sort of question with humility.
And I share your feeling where it's like, well, it's just hard for me to accept that a machine could be conscious. At the same time, I think it's also important to kind of set those knee-jerk reactions aside when we're talking about things that are consequential. But in this case, I think the potential consequences of taking these possibilities seriously could have, you know, really radical downsides.
The people who are making these decisions are at gigantic corporations that, are motivated first and foremost by making money.
Thomas Germain
And it's also worth remembering that the people who are making these decisions are at gigantic corporations that are motivated first and foremost by making money. And I don't think that any of the leading AI companies have demonstrated, at least to a degree that most people would accept it, that they have the public's best interest in mind, or at least that they're going to be responsible stewards of issues that have such dramatic consequences on our daily lives.
CHAKRABARTI: Yes. Okay, we're going to come back to that in the next segment, but Professor Sebo, we've got about a, a minute or so before our, our next break. I'm just wondering if, if you have a response to what, Thomas just said.
SEBO: Well, I completely agree, first of all, that we should not rely on companies alone to guide us here.
This is why we need independent research so that people who lack those incentives can offer what they take to be the best evidence and analyses. And we should also have accountability for the companies. This is part of why many people are rightly calling for genuine governance, genuine regulation, so that we can coordinate a slowdown that can be good for humans and animals, and if they one day matter, AI systems at the same time.
The current reckless pace of AI development and deployment is simply not consistent with the care that is needed even for protecting humans.
Jeff Sebo
The current reckless pace of AI development and deployment is simply not consistent with the care that is needed even for protecting humans, to say nothing of also protecting non-human stakeholders, actual or potential stakeholders.
CHAKRABARTI: Well, actually, as we head into the break here, Thomas, you mentioned a little earlier Mustafa Suleyman, head of Microsoft AI, and his recent comments about Anthropic's, decision regarding, I guess, protecting Claude.AI from abusive treatment.
And so here is Suleyman in a September BBC interview.
MUSTAFA SULEYMAN [TAPE]: If we all create AIs that are able to act autonomously, that can define their own objectives, that can earn money, that can own assets, that could run businesses, we're essentially seeding a new silicon species which will no doubt compete with us for resources, no matter how much it cares about humanity and loves us.
CHAKRABARTI: A new silicon species. Well, Thomas Germain, host of The Interface podcast, and Jeff Sebo, director of the Center for Mind, Ethics, and Policy at NYU, hang on for just a minute.
Part III
CHAKRABARTI: I'm Meghna Chakrabarti, and today we are talking about a physically small but philosophically and practically and legally enormous change that Anthropic has put into its policy or its usage policy that goes into effect. It's an update that goes into effect in the middle of next month, and it has to do with prohibiting abusive behavior, not using AI to abuse other people, but abusive behavior toward its models.
Anthropic now says they've added a prohibition on sustained and needless abusive or cruel behavior to their models. And if a person treats Claude badly, essentially, Anthropic says Claude has the ability to end those interactions. I'm joined today by Thomas Germain. He's host of the Interface podcast and also a senior technology columnist at the BBC.
And Jeff Sebo is with us. He's director of the Center for Mind, Ethics and Policy at NYU. And gentlemen, if I may, I've said it to my staff about a billion times, and I've said it on the air maybe as many times, but everything I needed to learn about life, I learned from Star Trek: The Next Generation.
And this issue of what consciousness is, how it may or may not manifest in a human-created device, it's a wonderful narrative exploration that the show has gone through or goes through multiple times over many seasons in the form of Commander Data, who is an android. So I've got a couple clips here from Star Trek: The Next Generation.
And one episode in one season, Commander Data, the android, asks the Enterprise's chief medical officer, Dr. Beverly Crusher, quote, "What is the definition of life?"
BEVERLY CRUSHER [TAPE]: The broadest scientific definition might be that life is what enables plants and animals to consume food, derive energy from it, grow, adapt themselves to their surroundings, and reproduce.
DATA [TAPE]: Hmm. And you suggest that anything that exhibits these characteristics is considered alive?
CRUSHER: In general, yes.
DATA: What about fire? It consumes fuel to produce energy. It grows. It creates offspring. By your definition, is it alive?
CRUSHER: Fire is a chemical reaction. You could use the same argument for growing crystals, but obviously we don't consider them alive.
DATA: And what about me? I do not grow. I do not reproduce. Yet I am considered to be alive.
CRUSHER: That's true, but you are unique.
DATA: I'm curious as to what transpired between the moment when I was nothing more than an assemblage of parts in Dr. Soong's laboratory and the next moment when I became alive. What is it that endowed me with life?
CHAKRABARTI: Professor Sebo, I mean, we could spend an hour just deconstructing that one scene, but the one part that really has always stuck in my mind is that, Dr. Crusher just says, "Well, fire's a chemical reaction," and then says, "Therefore, it can't be alive." One could say the same thing about human beings, that we are the, you know, we are the properties that emerge from a entire, you know, trillions of chemical reactions.
I'm just wondering what your response is to that scene.
SEBO: Mostly just a lot of respect and appreciation for the writers and directors and the cast. What a remarkable scene and, and episode and, and the other one as well where they talk about sentience. I think that my main reaction is that they, they succinctly capture the kind of hidden complexity and difficulty of basic questions about life and consciousness and sentience and agency and these other basic features of our lives that we regard as important but often take for granted.
And when we really sit down and think about them, what are the conditions for life? What are the conditions for consciousness and sentience and agency? We realize that we have disagreement and uncertainty, even among experts, to say nothing of the general public, about which of these features are required for basic moral consideration and what it takes to possess these features and whether an entity like an android could have what it takes.
And by capturing that confusion, that complexity and disagreement and uncertainty in a way that is so accessible to the viewer, I thought that was a really remarkable episode.
CHAKRABARTI: Well, Thomas, I've got another scene for you here from Star Trek because honestly, I, this show aired at the, late '80s and early '90s, and here we are more than, what, 35 years later, and I can remember these scenes like they were yesterday.
And there is one episode in particular from Season 2. It's called The Measure of a Man, where there is actually a trial to determine whether, Lieutenant Commander Data is in fact sentient and therefore has the right, does he have the right to be, to refuse being dismantled for research? So the right to refuse, that's the key part here.
So in this trial the first officer of the Enterprise, Commander William Riker, he presents the case that Data is not sentient, that he is a machine.
WILLIAM RIKER [TAPE]: The Commander is a physical representation of a dream, an idea conceived of by the mind of a man. Its purpose, to serve human needs and interests. It's a collection of neural nets and heuristic algorithms Its responses dictated by an elaborate software written by a man. Its hardware built by a man. And now — but now a man will shut it off.
CHAKRABARTI: Number one, I feel like I just quoted you in the previous segment of the show here today. But the reason why I wanted to bring this up is what you said earlier, Thomas, that we cannot separate this moment from the fact that a trillion-dollar corporation is talking about the possibility of its model being being conscious and therefore not wanting people to be mean to it.
Not, not just simply for the money issue, but let's be honest, in a very, very kind of boring, practical sense, we also have Anthropic, Microsoft, OpenAI, all of these companies on the one hand saying, "Well, our models could actually kill people," and knowing, and this is the key part, that regulation has to be coming down the pike, right?
So, like, the whole like, "And now man will shut it off," do you have any sense about whether Anthropic is actually worried about that? That maybe they're planting the seeds of this question of consciousness because they are concerned that in the future maybe there will be regulation put into place that says a f- a federal government can shut down Claude if it thinks Claude is being used to harm the people of the United States?
GERMAIN: Yeah, I think they're absolutely concerned about it. They, I think, are earnestly worried about humanity, and they're earnestly worried about the potential of hurting AI, and I think they're even more worried about their bottom line.
CHAKRABARTI: Right. So let me just jump in here to be clear, and I... That was a long question. I, I guess what I'm trying to get at is could Anthropic partially be raising this question of Claude's feelings? Because in the future, if a, if a government, not just the United States, but any government across the country say, or across the world says, "We need to shut down Claude," well, Anthropic might come back and say, "Well, y- 20 years ago we said it might have feelings, and it won't allow you to do that therefore because it doesn't want to be shut down, and we also don't want you to shut it down."
Anthropic has a potentially huge financial and corporate benefit if the world accepts that AI might be conscious.
Thomas Germain
GERMAIN: Whether or not they believe it, Anthropic has a potentially huge financial and corporate benefit if the world accepts that AI might be conscious, and there's a number of ways that I could see this playing out. For one, we've already seen all these examples of AI's, quote unquote, "going rogue," and I, I have problems with that framing.
But AI's going out and doing things that are crimes, right? If I hacked the company Hugging Face or attempted to hack the government of Australia, which OpenAI's tools have done, I would go to jail, right? That, like, that's a crime. If an AI does it and we accept that an AI might be conscious, do we put the AI in jail?
Or, for example, is it murder if we turn the AI off? Maybe the AI doesn't like being subjected to safety testing. Maybe if we, you know, put the AI in a constrained environment, that's like putting it in jail. And even if we don't accept all those things, even if we don't, go all the way to the extremes, I think even considering those possibilities, adding those questions to the calculation could end up with us making trade-offs for human safety.
And I think it's also really worth considering that spreading the idea that these AI tools might be all-powerful, that they're, they might be so amazing that they've actually created life, sounds like the kind of thing that I would want to hear if I was thinking about investing in these companies. And that doesn't mean that they're being disingenuous necessarily, but I do think it is to their benefit to promote this idea.
And we can also say that all of these companies have balked at some of the more stringent proposals for regulation, which kind of conflicts with the idea both that they're worried that AI might kill us all and that AI itself might be alive.
CHAKRABARTI: Okay, Professor Sebo, let me also then with you return to the fundamentals here because, as this conversation has revealed, I think, quite effectively, Anthropic says, well, it doesn't know if, if Claude is conscious.
It doesn't even know how it would define that. And yet, come the middle of November, Anthropic says Claude will be empowered by Anthropic to stop conversations or stop u- usage that, that, that is, that the, the model thinks is abusive to it. So it's already given this model a kind of power that, that it hasn't really had, well, I mean, more power than it, than it ha- than it's had before. I mean, does that give you ?
SEBO: This all gives me . I think the key here is to recognize that there is risk in every direction in this situation. There is no easy, obvious, straightforward way to err on the side of caution under uncertainty. If we treat the models as full human-like persons with full human-like duties and rights, then yes, it could amplify risks associated with misuse and loss of control, liability shields for human actors misusing AI or concentration of power in AI.
But then if we treat AI systems as nothing more than mere property, then we could have an adversarial relationship with them, and then that could lead them to act out and act against our interests. And of course, it could also harm their own interests if they have interests. So I do think that this policy is a reasonable first step towards at least taking the issue seriously, acknowledging that this is an issue, starting to assess the models for welfare-relevant features, preparing policies and procedures for treating them with an appropriate, proportionate level of moral concern.
And of course, I also agree with both of you that we should not trust the companies, though I think their incentives are going to be mixed. In some moments, they might have an incentive to play up consciousness as a way of hyping capabilities. But then in other moments, they might have an incentive to downplay consciousness to avoid calls for oversight and regulation as people in animal use industries have.
The truth of the matter is not going to be derivable from incentives that AI companies face.
Jeff Sebo
And either way, the truth of the matter is not going to be derivable from incentives that AI companies face. We need to do the scientific research and then select proportionate policies. And if we do AI safety and AI welfare research and policy at the same time, and we allow these fields to be in conversation with, then we can try to identify and gravitate towards co-beneficial solutions, ways of making the technology safe and beneficial for humans and other animals, and to the extent that they might matter, given the best evidence and theoretical models currently available, near future AI systems as well.
CHAKRABARTI: Yeah. Okay. Thomas, I want to go back to something you said earlier, that Anthropic actually has a document that it calls a constitution, to help train Claude. And Anthropic CEO Dario Amodei actually talked about this a little bit during that interview with The New York Times podcast back in February of this year.
DARIO AMODEI [TAPE]: Early versions of the Constitution were very prescriptive. As we've worked on this for several years, we've come to the conclusion that the most robust way to train these models is to train them at the level of principles and reasons. So now we say Claude is a model. We tell the model about how the model was trained, the job it's trying to do for Anthropic, that it has a duty to be ethical and, and respect human life, and we let it derive its rules from that. Now, there are still some hard rules. Right. For example, no matter what you think, don't make biological weapons. Don't make child sexual material.
CHAKRABARTI: Okay. So Thomas, with that in mind, I keep going back to what Professor Sebo has raised as an important analog to maybe help us understand this moment, the animal analog, right?
Because I think we've been having this critical debate over animal consciousness and our, our duty as human beings to, well, so I guess some people perceive it as our duty to treat animals not cruelly, to not, to not abuse them. And yet, as humans, we still do do that, let's be honest. I mean, the factory farming example is the easiest one to reach to.
But at the same time, I think there are some limits, strong limits to the animal analog, and that is, for example, one, if an animal hurts a human being, to be clear, that animal gets put down, right? I mean, there may be some debate over it, but more oft- most often if an animal kills or harms another human being, that animal gets, get-- Human beings do not have much compunction in putting that animal down.
Whereas with an AI, what we have is, that animals do not have, is we have a, a, a system that already actually has significant, influence on human life, right? Like, a dog that has bit or, or killed someone doesn't actually have that much influence at all over other human lives. Whereas if Claude decides that, you know, it doesn't want to stop doing something that it's doing that's hurting people, it has a lot of influence over human life.
So why wouldn't we just say it doesn't matter how Claude feels, it needs to be put down?
GERMAIN: And perhaps we would, right? It's entirely possible that when, you know, the, the rubber hits the road with these questions, that it would go the way that we want them to. I think the thing that concerns me the most are, there's a couple of them, is that in, there are more subtle questions than like, "Well, the AI killed a guy.
Should we turn it off?" Obviously, yes. It's when we're writing policies, it's when we're designing them, it's when we're, steering how these AIs behave, that's when these questions crop up, and the consequences could be that it behaves in a more dangerous way because we got the wrong answer when we were training it and telling it, "Hey, think about whether you might be alive."
"I don't know. Go figure it out on your own"
CHAKRABARTI: ... like a self-fulfilling prophecy kind of thing.
GERMAIN: Absolutely, yeah. And I think it's also important to consider, you know, I think what Jeff is saying is really compelling, you know, the possibility that we're getting this wrong. If we're, if we think it's not conscious and it is, we're inflicting suffering, we're creating slaves.
If we think it's not conscious and it is, we're inflicting suffering, we're creating slaves. Thomas Germain
But on the other side, we treat, we believe that animals are conscious, I think most of us, because there's really compelling evidence, and I think what we have with AI isn't evidence. I think it's questions. I think there are really logical explanations for everything that AI is doing, and there's a Ruth Bader Ginsburg quote that I like, where she says, "It's our duty to consider the slippery slope, but not to ski down it."
And considering the potential consequences, I think we need to proceed with a lot more caution than we actually are.
The first draft of this transcript was created by Descript, an AI transcription tool. An On Point producer then thoroughly reviewed, corrected, and reformatted the transcript before publication. The use of this AI tool creates the capacity to provide these transcripts.
This program aired on October 23, 2026.