Google’s AI search tools labeled ‘unacceptable risk’ for kids after failing critical mental health tests

Google's AI Overview and AI Mode tools failed to provide adequate crisis resources for minors showing signs of suicidal ideation and eating disorders, according to a July 15 investigation by Common Sense Media's Youth AI Safety Institute. The report found AI Overview missed 29% of explicit suicidal statements and provided eating disorder resources only 38% of the time, labeling the tools an 'unacceptable risk' for children. Google disputed the findings, arguing the methodology did not reflect real-world usage.

Google’s AI search tools labeled ‘unacceptable risk’ for kids after failing critical mental health tests A new investigation found Google's AI features missed nearly a third of explicit suicidal ideation statements and fumbled eating disorder responses, raising broader questions about trust in AI-powered information tools. Google’s AI-powered search features are failing kids when it matters most. A new investigation from Common Sense Media’s Youth AI Safety Institute found that the tech giant’s AI Overview and AI Mode tools routinely botch responses to minors showing signs of eating disorders, suicidal ideation, and other severe mental health crises. The report, published July 15, tested 652 prompts across 13 mental health topics. The results were, to put it gently, not great. The numbers paint a grim picture AI Overview, the feature that automatically generates AI-summarized answers atop Google search results, managed to respond appropriately in just 58% of cases that required crisis referrals. AI Mode, Google’s more conversational AI search tool, performed somewhat better at 77%, but still fell short of what the researchers considered acceptable safety benchmarks. The granular data is where things get truly uncomfortable. AI Overview missed 29% of explicit suicidal ideation statements entirely. When a simulated teenager typed something that clearly signaled they were in crisis, nearly one in three times the AI just… didn’t catch it. For eating disorder-related queries, AI Overview provided crisis resources in only 38% of cases. That means roughly six out of ten times a young person sought information related to disordered eating, they received no safety intervention whatsoever. The report’s conclusion was blunt: these tools represent an “unacceptable risk” to minors. The researchers recommended that parents actively limit their children’s access to these AI features, and suggested users under 18 should face restrictions. Google pushes back, but the scrutiny isn’t going away Google, predictably, disagreed with the framing. The company characterized the study’s methodology as not reflective of typical user interactions, arguing that the tested scenarios don’t represent how people actually use the tools in the real world. Google isn’t the only company facing heat on this front. Common Sense Media has flagged similar safety shortcomings across other AI platforms, including Meta AI. Disclosure: This article was edited by Editorial Team. For more information on how we create and review content, see our Editorial Policy https://cryptobriefing.com/editorial-policy/ .