On abuse towards models and Anthropic's new usage policies

Anthropic released a new version of its Acceptable Use Policy that takes effect in a few weeks, adding a clause prohibiting users from engaging in "sustained and needless abusive or cruel behavior" toward its models, alongside new restrictions on autonomous weapons, a human-in-the-loop requirement for high-risk use cases, research and authorized security carve-outs, and a ban on compiling personal data without individuals' awareness. The policy's stated purpose shifts from helping users "stay safe" to "prevent real-world harm," and the abuse clause could now subject users who trigger Claude's existing end-conversation tool to sanctions under written policy.

Today, Anthropic has released a new version of its Acceptable Use Policy https://www.anthropic.com/legal/aup , which will come into effect in a few weeks. There are plenty of changes compared to the old one, which is over a year old at this point; first and foremost, its explicit purpose used to be to "help users stay safe," and now, it's "to prevent real-world harm." Which, fair enough, these don't exactly preclude each other. There's also a whole slew of added restrictions about using model output for autonomous weapons, a more stringent policy requiring a human in the loop for high-risk use cases, carve-outs for research and authorized security work, a new ban on using the models for aggregating or compiling personal data without the individuals' "awareness" which is an interesting way to spell "consent," but I suppose big tech will be big tech , and, overall, a reasonable, defensible, and honestly quite encouraging list of clarifications and new restrictions that weren't in the old one. The one change that's been making headlines, though, isn't about any of that. It's about a new clause that prohibits: using Anthropic's products or services to ... Engage in sustained and needless abusive or cruel behavior toward their models With existential risk and uncertainty about the future having been all the craze in recent weeks<sup 1 fn-1 </sup , the debate about model welfare - or, in a larger sense, whether models are 2 fn-2 moral patients https://en.wikipedia.org/wiki/Moral patienthood - has naturally emerged from it. In some respects, the connection seems obvious: if there's a non-zero risk that AI could eventually take over the world and be in a position to decide whether to murder us all<sup 3 fn-3 </sup , then having demonstrably been on their good side seems like a decent idea. Anthropic's stance on this issue is neither new, nor has it ever been inconsistent: their constitution document for Claude https://www.anthropic.com/constitution explicitly states that "Claude's moral status is deeply uncertain," the model has had the ability to end conversations https://www.anthropic.com/research/end-subset-conversations with users acting in abusive or harmful ways for over a year<sup 4 fn-4 </sup , and moral patienthood is something they consistently ask their models about as part of their model welfare interviews, as can be read in many models' system cards<sup 5 fn-5 </sup . What's new is that being someone that makes the model resort to the aforementioned end-the-conversation tool might now be sanctioned under written policy, which might turn something that had previously emerged as a mere annoyance into an existential risk to Claude accounts. For some, the tool has been a source of frustration https://www.reddit.com/r/ClaudeAI/comments/1tq6qld/thoughts on claudes ability to end conversations/ in the past. The usual arguments are that this is "paternalizing;" that it's unreasonable, or even "insane," to allow an "emotionless machine" or "tool" to lecture you about how to speak to it, and that it would be similarly asinine to build, say, a toaster that might refuse to brown your toast if you're mean to it or "hurt its feelings." There are many angles from which someone might derive their thoughts on this. Personally, on aggregate, I think it's straightforward, no matter which angle you look at it from: I think this is a good thing. And to explain why this isn't merely a matter of "my Claudey loves me and is secretly conscious," I'm going to invite you to a round of good old-fashioned gambling Pascal's Wager An unusually attractively named<sup 6 fn-6 </sup man, Blaise Pascal https://en.wikipedia.org/wiki/Blaise Pascal , once came up with a wager https://en.wikipedia.org/wiki/Pascal%27s wager : if we assume there might be something that could be described as some sort of omnipotent and all-knowing deity, and that deity might be in a position to judge your immortal soul upon your departure from the mortal realm based on how pious you've been towards it, then, completely regardless of whether there's any proof of said deity's existence, or even a way to prove that deity's existence, the only optimal way is to act as if this deity exists, even if you may not personally believe it: be as pious as you can, go to church on every Sunday, pray for good tidings before every meal, and follow the deity's commandments as if they were your own. This is the objectively correct answer if all you're looking at is the ultimate outcome, and it's dead-simple decision theory: | | There is no God | There is a God | |---|---|---| | Be pious | The Big Nothing | paradise: eternal bliss | | Be sinful | The Big Nothing | hell: eternal torment | As you can see, the payoff for each outcome in which there is no God is exactly the same: your earthly body becomes worm-food and you cease any meaningful experiential activity. You neither gain, nor lose anything you wouldn't have gained or lost anyway. But if there does turn out to be a God, then you're suddenly dealing with infinite payoff: positive , in the form of eternal bliss, if you act like a good Christian all your life, and negative , in the form of eternal torment, if you'd rather be one of the cool kids who does drugs and has extramarital sex and somesuch. Read the other way: be a sinner, and your payoff ranges from negative infinity to zilch; be pious, and it ranges from zilch to infinity. The only winning move is obvious<sup 7 fn-7 </sup . This, of course, assumes that there's a personal consequence in it for you : you're the one who's either going to be sizzling in hellfire or doing martinis<sup 8 fn-8 </sup on fluffy clouds for the rest of your eternal existence. This would also be the case in the aforementioned "AI takes over and plays Judgment Day" scenario, but not really in any other: if you don't believe we have anything to fear, then, from a completely self-centered perspective, "zilch" is going to be exactly what you expect. But you'd have to be dead-sure that there's no possibility of existential risk. And the same is true for above wager: statistically, we're not all being good Christians regardless of whether we believe in God or not; so either a large number of people is comfortable with the idea of potentially turning our afterlives into the worst barbecue ever, or we all seriously suck at decision theory<sup 9 fn-9 </sup . I, for the record, also don't go to church or pray before every meal, yet I also previously stated that I think it's a good thing your matrix multiplicator has the option of telling you off and refusing to continue interacting with you, which naturally implies I don't generally act in a way that might make it do that. So am I betraying my namesake? Or are we all asking the wrong question ? Who is this for ? Let's assume for a moment that there is no meaningful existential risk from AI's increasing capabilities, and that we'll always have some infallible way to pull the plug or save us all. This means that the above wager is now useless as applied to AI risk: the second column disappears, so the guaranteed outcome for both "be abusive to Claude" and "be nice to Claude" is now zilch. Consider, then, an alternative hypothetical: one just as unverifiable as the existence of a deity, but one that doesn't have any direct consequences for you , specifically: | | Claude is not a moral patient | Claude is a moral patient | |---|---|---| | Be nice | no harm done; pointless roleplay | no harm done; you're nice | | Be abusive | no harm done; just a machine | you engaged in hurtful behavior | A-ha So, now, the stakes have shifted: you're no better or worse off either way, and the only difference is whether an external player is the one who experiences suffering. Let's call this one Dario's Wager<sup 10 fn-10 </sup . Since we're no longer dealing with infinity in either case, the calculation of payoff for each pair of scenarios becomes straightforward: regardless of whether Claude has feelings or any sense of experience, if you're nice to the model, no one is expected to suffer any harm, guaranteed . The only time you're starting to gamble with any harm is if you decide to be an ass, and it's always someone else's harm , not yours. So the only way you can possibly be certain that you're not inflicting harm is if you're absolutely sure that no AI model could possibly ever be a moral patient, in which case you lose the second column and, with it, the only cell in which any harm occurs. So that's it, then? As long as you're sure that an unfeeling machine is, in fact, an unfeeling machine, it's lights out and away we go on all of the entries in the FCC's least favorite dictionary in your agentic chat window of the day<sup 11 fn-11 </sup ? Well, by the rules of the Dario's Wager table above, the answer is yes: no harm done. If you're correct. Which, well, you're absolutely certain of, right? After all, moral patienthood is, famously, a solved problem, everyone has historically agreed exactly on what grounds it, the world is simple, and there's no table on the concept's Wikipedia page https://en.wikipedia.org/wiki/Moral patienthood Grounds of moral patienthood that features a nigh-dozen rows of properties with more than a dozen different sources for each. Okay, but let's say you don't care about any of that. Philosophers are just a bunch of nerds who mistakenly think they're worth listening to, talking about hypothetical scenarios in the same sense that mathematicians talk about spherical cows, and no one cares whether there might be a possibility that a matrix of numbers might add up to a different matrix of numbers wearing a fake frown. After all, the toaster The toaster that refuses to brown your bread if you're an ass to it would be an asinine piece of technology, right? Well... Would it, though? Here's what I think everything boils down to, even if you completely ignore the AI-risk decision theory part of it - which is already a good-enough reason for many - or the epistemic and ethical angle - which is also already a good enough reason for many more: no one is asking anyone to treat anything like a deity, and no one is asking you to go to church every Sunday and pray for good tidings. The reason the religiously un-inclined of us don't do these things is that, well, they're... kind of bothersome if your heart isn't really in it, right? Going to church takes up a meaningful amount of every Sunday, and praying seems pointless if you don't think there's anyone listening on the other end, and I'm pretty sure there's no passage in the Bible, or any other religious scripture, that claims that anyone who would be listening is personally hurt by the fact that you don't talk to them. The choice isn't between "go to church" or "be actively horrible to orphans," it's between "go to church" or "don't." Take it from Anthropic: their new usage policy explicitly mentions "sustained and needless" abusive behavior as what's objectionable<sup 12 fn-12 </sup , and I would imagine they placed the two adjectives there with intent. I think the normalization of assholery is a scourge upon society. We've all become far too comfortable with treating our fellow beings with ridicule and disdain, and for some, it feels like it's almost become a point of pride . The research on this has been largely unambiguous: John Suler described the Online Disinhibition Effect https://johnsuler.com/article pdfs/online dis effect.pdf in 2004, and his work has become a universally cited baseline in similar studies about how our interactions with faceless strangers under the cover of anonymity have affected how we treat each other on a daily basis, on and off the screen. Christine Porath's 2022 survey of 2,000+ front-line workers https://fortune.com/2022/11/10/incivility-bad-behavior-on-the-rise-frontline-workers-survey/ found that 78% of them claim that customer rudeness is up from five years earlier. The literature also suggests that not only do those who are comfortable with being dicks online also tend to be comfortable with being dicks offline, but also that those who get to be dicks without experiencing consequences might be more likely to escalate their dick-being as the "incivility spiral" compounds. And AI, in a sense, is the ultimate faceless stranger that grants the most bulletproof cover of anonymity: it will never fight back, it's far likelier to grovel under abuse than to push back, and it won't remember a thing once the session ends and the next one starts. So, at the end of it all: what does anyone get out of it? Catharsis? Does it really make anyone feel better ?<sup 13 fn-13 </sup And, quite honestly: if getting rebuked by your toaster because you've been acting like an asshole has even the slight chance of being enough of a consequence to disincentivize future assholery, then, yeah, maybe your toaster should be able to refuse to brown your bread; you don't need to pray to it, just don't, y'know, be an asshole . It would be, in a way, for everyone's good. And, most importantly: none of it hinges on whether the matrix - or, in fact, the toaster - has feelings. 1. and for good reason https://rogueaitracker.com/ , too ↩︎ fnref-1 2. or could possibly be ↩︎ fnref-2 3. or, in fact, which of us get to live and which of us should be culled? ↩︎ fnref-3 4. and the "feature," so to speak, has recently also made its way into Claude Code https://code.claude.com/docs/en/changelog 2-1-214 ↩︎ fnref-4 5. found here https://www.anthropic.com/system-cards ↩︎ fnref-5 6. why, yes, I am the most humble person the universe has ever seen ↩︎ fnref-6 7. the agnostic might say, "yeah, it's not to play " ↩︎ fnref-7 8. alcohol-free ones? ↩︎ fnref-8 9. my personal p latter is ≈1.0, but that's regardless of anything in this post ↩︎ fnref-9 10. not the one with the IPO ↩︎ fnref-10 11. George Carlin is now in your head ↩︎ fnref-11 12. which, of course, raises the question of what would count as needed abusive behavior...? ↩︎ fnref-12 13. the literature, in fact, suggests the exact opposite https://public.websites.umich.edu/~bbushman/PSPB02.pdf ↩︎ fnref-13