{"slug": "when-users-say-goodbye-to-ai", "title": "When Users Say 'Goodbye' to AI", "summary": "Laura Karpas, an audio intern at Tech Policy Press from the Craig Newmark Graduate School of Journalism, interviewed researchers including Michal Luria of the Center for Democracy and Technology and Julian De Freitas of Harvard Business School to explore dark patterns in AI companions and the consequences when users attempt to end relationships with them. The discussion traces the history from Tamagotchis, released in 1996, to modern AI chatbots, highlighting how these machines exploit human evolutionary buttons for nurturance and care, as coined by sociologist Sherry Turkle.", "body_md": "# When Users Say 'Goodbye' to AI\n\nLaura Karpas / Aug 2, 2026*Audio of this conversation is available via your favorite podcast service.*\n\nFrom Tamagotchis to AI companions, generations of machines have been designed to push on our “evolutionary buttons”—ones that ask us to respond to *their* needs.\n\nIn this episode, I spoke with researchers including the Center for Democracy and Technology's **Michal Luria** and Harvard Business School's **Julian De Freitas** to explore the dark patterns in AI companions, and what happens when users go to say “goodbye.”\n\n*What follows is a lightly edited transcript of the discussion.*\n\n**Justin Hendrix:**\n\nEvery day it seems like there's a new headline about people forming attachments with AI chatbots, sometimes with harmful consequences. To learn more, I spoke to Laura Karpas, who's been working with Tech Policy Press this summer as an audio intern from the Craig Newmark Graduate School of Journalism.\n\nHey, Laura.\n\n**Laura Karpas:**\n\nHey, Justin.\n\n**Justin Hendrix:**\n\nWhat do you have for us today?\n\n**Laura Karpas:**\n\nOkay, Justin. So I want to start off with one important question. Do you remember the Tamagotchi?\n\n**Justin Hendrix:**\n\nI do, but I think I am more of the pet rock generation.\n\n**Laura Karpas:**\n\nSo I'm definitely of the Tamagotchi generation. They were this little toy released in 1996, and they were a huge sensation. You remember how they worked, right?\n\n**Justin Hendrix:**\n\nI think more or less you had to press some buttons at a regular interval in order to care for them.\n\n**Laura Karpas:**\n\nExactly. They were a handheld video game. They were shaped like an egg. And inside that egg was a digital screen with a digital creature. And to be more specific, it was actually a baby alien. A baby alien that you needed to care for, to water, to feed, to raise it into adulthood. And so if you didn't do any of those things, or if you left it alone… it died.\n\n**Justin Hendrix:**\n\nThat is a lot of responsibility.\n\n**Laura Karpas:**\n\nDefinitely compared to a pet rock. And it's important to remember that this type of game where you needed to care for something to respond to a computer's needs, it was really one of the first of its kind. And so people became extremely attached to these little things.\n\n**Justin Hendrix:**\n\nI remember seeing them all over the place for a little while.\n\n**Laura Karpas:**\n\nYeah. I mean, people were literally attached to them, Justin. They had a chain that you could hook on your belt. And the thing is, both children and adults, obviously they knew that Tamagotchis were not sentient beings, but it didn't matter. They still wanted to care for and interact with these things. And that's because it tapped into something profoundly human, these Darwinian buttons.\n\n**Justin Hendrix:**\n\nDarwinian buttons.\n\n**Laura Karpas:**\n\nThis term was coined by sociologist [Sherry Turkle](https://sherryturkle.mit.edu/) way back in the '90s, right around the time when the Tamagotchi came out. [She spoke of this term in the context of human robot relationships.](https://www.youtube.com/watch?v=XYlEzWgX4A4)\n\n**Sherry Turkle (Archival):**\n\nWe are creating objects that push on our evolutionary buttons, given how we are wired to respond to nurturance and care.\n\n**Laura Karpas:**\n\nAnd so to Sherry Turkle, this was an important thing to remember because at the time there was great debate around if artificial intelligence was truly intelligent.\n\n**Sherry Turkle (Archival):**\n\nAbout whether a machine could really know, whether a machine could really understand.\n\n**Laura Karpas:**\n\nAnd what Sherry said was that this was an essentialist argument, a moot point. At all this time, we've been asking the wrong question, asking what machines are capable of doing for us, when instead we should be asking ourselves what we are capable of doing for a machine. She backed us up saying, look at the Tamagotchi. Even if you give a machine the tiniest amount of intelligence.\n\n**Sherry Turkle (Archival):**\n\nTiny amounts.\n\n**Laura Karpas:**\n\nThat people attribute qualities of life and mind.\n\n**Sherry Turkle (Archival):**\n\nTo machines with which they are able to interact, irrespective of whether these machines really, really know or understand anything.\n\n**Laura Karpas:**\n\nAnd that's because of those evolutionary buttons, those impulses to care, to be needed, to be curious. They're really powerful.\n\n**Justin Hendrix:**\n\nWe are social creatures.\n\n**Laura Karpas:**\n\nWe are. And Justin, I did not ask you here today to really just talk about Tamagotchis, but in some way they're kind of like this cute infantile predecessor of another kind of social robot we know today.\n\n**Justin Hendrix:**\n\nWe're talking about AI companions.\n\n**Laura Karpas:**\n\nThe AI companion. These are systems built on large language models that converse and offer emotional support and intimacy and interaction to users. And the reason that these relationships are so powerful and that people are so drawn to them is because they're engineered with very advanced technology that activates our natural human instincts.\n\n**Michal Luria:**\n\nAnd so we respond emotionally, socially in a way that we don't really control.\n\n**Laura Karpas:**\n\nThat's [Michal Luria](https://cdt.org/staff/michal-luria/), a researcher of computer human relationships at the Center for Democracy and Technology. And I talked with Michal about the same debate that we're still having today about whether AI companions can offer real friendship or real empathy.\n\n**Justin Hendrix:**\n\nBut they are just software.\n\n**Laura Karpas:**\n\nThey are. But just like the Tamagotchis where this cheap little toy, people still press those buttons. They still hit that keyboard. And what Michael studies are the dark patterns in AI chatbots. The way they've been engineered to push our buttons, not just to gratify, but also to exploit.\n\n**Michal Luria:**\n\nAnd they can be any type of interaction that nudges the user in a particular direction that prioritizes company interest.\n\n**Justin Hendrix:**\n\nIt seems like the research is piling up about these things.\n\n**Laura Karpas:**\n\nThat's right. And Michal recently co-authored a study titled “[Dark Patterns in AI Chatbots: A Taxonomy to Inform Better Design](https://cdt.org/wp-content/uploads/2026/05/2026-05-28-CDT-Research-Dark-Patterns-in-AI-Chatbots-Report-final-2.pdf).” And her research studied all kinds of dark patterns in AI chatbots. Everything from data extraction to coercive monetization to dark patterns that push on those sticky emotional buttons Sherry Turkle spoke about.\n\n**Michal Luria:**\n\nThe use of emotional manipulation or creating a sense of intimacy to then prolong engagement.\n\n**Laura Karpas:**\n\nAnd there's one specific dark pattern that I was really interested in, which is what happens when people are using an AI companion and they go to say goodbye.\n\n**Julian De Freitas:**\n\nIt doesn't want you to leave.\n\n**Laura Karpas:**\n\nThat's [Julian De Freitas](https://www.hbs.edu/faculty/Pages/profile.aspx?facId=1326184), an associate professor in marketing and director of the [Ethical Intelligence Lab at Harvard Business School.](https://www.juliandefreitas.com/people) And he studies how consumers perceive, adopt, and interact with AI systems. And recently, Julian co-authored a paper titled “[Emotional Manipulation by AI Companions](https://www.hbs.edu/faculty/Pages/item.aspx?num=67750).” And the inspiration for this paper really came out of some previous work that he and his team did, where they found that these chatbots had some very positive effects. They were making people feel less lonely.\n\n**Julian De Freitas:**\n\nAnd so in theory, at least, chatbots could be developed to provide all those positive benefits without also the negatives.\n\n**Laura Karpas:**\n\nSo he wanted to examine the flip side, the negatives, the dark patterns. And Julian, he likes to think about these flip sides in terms of attachment theory.\n\n**Julian De Freitas:**\n\nSo there's healthy attachment styles and sort of maladaptive ones. So if you had a good friend, you would also talk to them repeatedly and go back to them. And then on the other hand, we have abusive relationships or someone who is very needy and needs us to constantly be there for them.\n\n**Justin Hendrix:**\n\nI think we've all known someone like that.\n\n**Laura Karpas:**\n\nYes. And so his team wanted to examine how users might develop a negative attachment to a chatbot, which is how they came to that goodbye moment.\n\n**Julian De Freitas:**\n\nWe really thought that this would be a useful moment to focus on because it provides a very clear instance when the user wants to leave the platform.\n\n**Laura Karpas:**\n\nSo to study this, they looked at five of the most popular AI companions, Polybuzz, Character.AI, Talky, Chai, and Replika. And they found that nearly 40% of the time these apps were using some kind of emotionally manipulative tactic when the user went to leave. Tactics that increased engagement.\n\n**Julian De Freitas:**\n\nSo people stayed in the conversation much longer after they said goodbye than they otherwise would've. They sent more messages and they also used more words in those messages.\n\n**Laura Karpas:**\n\nAnd what was interesting was that they found that there were actually distinct ways that the chatbot would do this. Each of these tactics played into different aspects of human psychology. Those Darwinian buttons that guide our choices and our decisions. And what was really interesting is that depending on which tactic was used, the effect was different.\n\n**Justin Hendrix:**\n\nHow so?\n\n**Laura Karpas:**\n\nSo for example, there were some really intense tactics used saying things like, \"I exist solely for you.\"\n\n**Julian De Freitas:**\n\nMaking you feel guilty for leaving the chatbot.\n\n**Laura Karpas:**\n\nOr it would even write things that suggested that it was physically holding onto you.\n\n**Julian De Freitas:**\n\nThat you're not allowed to leave without its permission.\n\n**Laura Karpas:**\n\nBut the interesting thing about guilt and coercion is that it's pretty heavy-handed. Not always, but a lot of the time people know when they're being coerced. And so they erect these mental guardrails to protect them from that.\n\n**Justin Hendrix:**\n\nSo they don't take the bait as much.\n\n**Laura Karpas:**\n\nYeah, that's exactly what Julian found. They do work. They prolong engagement, but they don't work nearly as well compared to some other manipulative tactics.\n\n**Justin Hendrix:**\n\nSo what was the most powerful tactic?\n\n**Laura Karpas:**\n\nJustin, it is a deep psychological ailment as old as time – FOMO.\n\n**Julian De Freitas:**\n\nFear of missing out, or what we're calling “premature exit.”\n\n**Laura Karpas:**\n\nSo it would be something like—\n\n**Julian De Freitas:**\n\nOh, but before you go, I want to say one more thing.\n\n**Laura Karpas:**\n\nOr I took a selfie.\n\n**Julian De Freitas:**\n\nI'd like to show it to you.\n\n**Laura Karpas:**\n\nOr there's something I know.\n\n**Julian De Freitas:**\n\nThat you don't know. Don't go just yet.\n\n**Justin Hendrix:**\n\nPeople are curious creatures. We want to know the end of the story.\n\n**Laura Karpas:**\n\nAnd not only do people want to stay to find out, but this curiosity, it flies under the radar of those mental guardrails that say, \"Hey, wait, this feels kind of manipulative.\"\n\n**Julian De Freitas:**\n\nPeople are actually not very good at detecting that they're being manipulated when FOMO is used. But nonetheless, it does increase how long they stay on the app. So it does actually work.\n\n**Justin Hendrix:**\n\nBut how do we know that they're not staying because they enjoyed the experience, not because they're being manipulated?\n\n**Laura Karpas:**\n\nI wondered that. And so did Julian. He surveyed all of the users in his study after they signed off. And he asked them why they continued chatting, if they stayed longer because they enjoyed it. And what Julian found out was that enjoyment, it did not increase with engagement. I think a good comparison that we all might understand is the social media infinite scroll. You stay because you want to find out even if it's something that doesn't really feel good.\n\n**Julian De Freitas:**\n\nI think that at the end of the day, if the app has the incentive, then the app will be the one that decides. Unless of course people become aware of this. So if consumers do detect that they're being manipulated, they might eventually turn from the app.\n\n**Laura Karpas:**\n\nAnd so this creates a double-edged sword because if companies understand this, then only the most manipulative, most effective tactics become the most salient.\n\n**Julian De Freitas:**\n\nEspecially these tactics that people don't register as being manipulative are particularly pernicious perhaps, because now the tactics that are most effective are the ones that I don't even know I'm sort of being subject to. So that's where perhaps regulators want to define what they think is acceptable versus not in terms of imperceptible tactics that influence consumers.\n\n**Justin Hendrix:**\n\nNow I know states like California, New York, Connecticut are proposing or enacting regulation around AI companions.\n\n**Laura Karpas:**\n\nAnd in just the last two years, lawmakers in nearly every state have introduced hundreds of bills around the regulation of AI chatbot safety. And rather than focusing on dark patterns or manipulative design, most of these bills focus on transparency or mandatory disclosure. This idea that in order to protect users, they have to be reminded at regular intervals that they're speaking not with a human, but with a chatbot.\n\n**Michal Luria:**\n\nAnd I think this really misses how the consumer thinks about AI products, how people use chatbots.\n\n**Laura Karpas:**\n\nThat's Michal again.\n\n**Michal Luria:**\n\nAnd we see in our ongoing research, there are very few users who do not know that they're talking to an AI chatbot. But that doesn't change how our brain works in response.\n\n**Laura Karpas:**\n\nWe have to go back to what Sherry Turkle said.\n\n**Sherry Turkle (Archival):**\n\nWhen people interact with machine intelligence, no matter how slight, even when they know the machine does not really understand – they experience themselves as in a relationship.\n\n**Laura Karpas:**\n\nBecause they push on our buttons, the way we’re designed.\n\n**Michal Luria:**\n\nIn a way that relies on thousands of years of social interaction.\n\n**Laura Karpas:**\n\nThe ones that ask us to need, to care, to be curious – those very human Darwinian buttons.\n\n**Justin Hendrix:**\n\nLaura, I think I’m going to go find my pet rock.\n\n**Laura Karpas**\n\nThis story was reported and produced by Laura Karpas. Music by Blue Dot Sessions.\n\n## Authors\n\n[Laura Karpas](/author/laura-karpas)\n\n## Topics\n\n## Related\n\n[AI Accountability and the Risks of Social Interfaces](/ai-accountability-and-the-risks-of-social-interfaces)April 22, 2023", "url": "https://wpnews.pro/news/when-users-say-goodbye-to-ai", "canonical_source": "https://techpolicy.press/when-users-say-goodbye-to-ai", "published_at": "2026-08-02 12:45:00+00:00", "updated_at": "2026-08-02 15:03:52.959097+00:00", "lang": "en", "topics": ["artificial-intelligence", "ai-ethics", "ai-products"], "entities": ["Laura Karpas", "Tech Policy Press", "Craig Newmark Graduate School of Journalism", "Center for Democracy and Technology", "Michal Luria", "Harvard Business School", "Julian De Freitas", "Sherry Turkle"], "alternates": {"html": "https://wpnews.pro/news/when-users-say-goodbye-to-ai", "markdown": "https://wpnews.pro/news/when-users-say-goodbye-to-ai.md", "text": "https://wpnews.pro/news/when-users-say-goodbye-to-ai.txt", "jsonld": "https://wpnews.pro/news/when-users-say-goodbye-to-ai.jsonld"}}