The company is also building the ability to contact emergency services if a user may be at imminent risk of harming themselves.
Meta is expanding its safeguards for teen users in an effort to show parents, and perhaps regulators, that its platforms are safe for them to use. A few months after it started showing guardians the topics of their teens' AI conversations, Meta has introduced a new safety protection feature that will also let them know if their children's chats suggest they're considering self-harm or suicide.
When an underage user talks about suicide or self-harm, Meta AI already directs them to crisis helplines and encourage them to reach out to an adult who can help. Starting today, the company will also proactively notify the teen's supervising parents or guardians that they're at risk. It will share resources and advice on what they can do for their children, as well.
Meta says it built a dedicated AI system to identify these conversations, but it will manually review all chats flagged by the AI before it sends out an alert in order not to cause panic when there's no real cause for it. However, it will err on the side of caution and send out the notification even if the teen user's intent seems ambiguous to the human reviewer. This feature is now live for parents using Instagram parental supervision in the US, UK, Australia and Canada. It will be available to everyone using parental supervision around the world by the end of the year.
The company is also working on building the ability to contact emergency services if a user's conversation with Meta AI suggests that they may be at imminent risk of taking their own life, whether they're a minor or an adult. In May, OpenAI rolled out a feature that also intends to address self-harm. Called Trusted Contact for ChatGPT, it allows users to nominate a friend that the company can contact if they're at risk of harming themselves.
Finally, Meta's Limited Content setting now also applies to Meta AI interactions. If a parent has switched on Limited Content on Instagram, which filters out content with sensitive topics like mature visuals and self-harm, their children's AI interactions will also be more limited.
If you or someone you know is experiencing suicidal thoughts, do not hesitate to contact the National Suicide Prevention Lifeline at 1-800-273-8255. The line is open 24/7 and there's also online chat if a phone isn't available.