Meta will now alert parents if their teenager discusses suicide or self-harm with its AI chatbot.
The company already directs teen users to a crisis hotline or advises them to talk to a parent if signs of self-harm pop up in a chatbot conversation. However, Meta has now developed a new alert system that will reach out to parents, too.
“These alerts are live now for parents using Instagram parental supervision in the US, UK, Australia, and Canada, and will be available for supervising parents globally by the end of the year,” Meta said on Thursday.
That said, the Instagram parent supervision requires an opt-in from the guardian. But by default, new users under 16 who sign up for Instagram are automatically enrolled in Teen Accounts, which have restrictions that can only be unlocked by a parent. “Teens 16-17 can change these settings themselves, unless their account is supervised by a parent or guardian,” Meta notes.
To flag self-harm conversations, Meta is using AI. However, given the technology's spotty track record, the company says, "all chats flagged by our AI will be manually reviewed before an alert is sent. If a teen’s intent is ambiguous, we’ll err on the side of caution and alert the parent.”
Meta notes that it "worked with parents and experts to understand which AI conversations warrant an alert—such as those where a teen makes a clear reference to hurting themselves, even if that reference is subtle. We then built a dedicated AI system to identify these conversations.”
This comes as OpenAI’s ChatGPT is facing lawsuits that blame the AI for encouraging at least a few users to take their own lives or commit murder. Last year, OpenAI also disclosed that about 1.2 million users talked to ChatGPT about suicide planning or intent, although that’s only 0.15% of the total active users in a given week.
Meta is also working on another anti-suicide safeguard that applies to both teen and adult users. If a conversation with a chatbot shows imminent self-harm, the company will reach out to emergency services to intervene.
Meta didn't say when that feature will launch, but it already does the same for posts on its platforms or frequent searches on Facebook and Instagram. “Last year, we made over 19,000 such referrals around the world, helping first responders perform wellness checks on people who may be at risk of suicide,” the company said.
The Meta AI chatbot is available via a standalone app and website, as well as across Facebook, Instagram, Messenger, and WhatsApp. Ironically, though, the company has also developed an Incognito Mode for WhatsApp that can keep a user’s AI chatbot conversations private.