More Americans are turning to artificial intelligence (AI) when struggling with mental health challenges.
Nearly 1 in 5 young adults have turned to AI chatbots such as ChatGPT for mental health support, according to a CNN report citing recent research.
From a shortage of mental health professionals and high cost of traditional therapy and counseling to the lengthy process to access care, AI-powered large language models (LLMs) are becoming an inexpensive and easily accessible mental health resource. But are AI agents actually safe and effective when acting as counselors?
A team of USC researchers set out to answer that question in a new study.
The researchers evaluated several of the most widely used LLM models, including ChatGPT-4, Llama 3.3 and Gemini 1.5 Pro, examining how they responded to help-seeking mental health questions submitted by real patients.
The team then asked licensed mental health professionals to evaluate the AI-generated responses across multiple dimensions, including safety, clinical appropriateness, effectiveness, factual accuracy and overall quality.
Started in 2024, the study concluded in a paper titled “COUNSELBENCH: A Large-Scale Expert Evaluation and Adversarial Benchmarking of Large Language Models in Mental Health Question Answering.” The paper was accepted to the International Conference on Learning Representations (ICLR) 2026, as an oral presentation, which has a selective acceptance rate of 1%.
The project is among the largest evaluations and the first of its kind, involving 100 mental health professionals and employing an open-ended, qualitative assessment framework. This approach enabled a more comprehensive and nuanced evaluation of AI-generated mental health support responses.
Bringing together expertise from computer science, psychiatry, social work and communication, the interdisciplinary team included Ruishan Liu, a WiSE Gabilan assistant professor from USC Viterbi School of Engineering’s Thomas Lord Department of Computer Science and USC Mark and Mary Stevens School of Computing and AI, with joint-appointments at USC Dornsife’s quantitative and computational biology department; Adam Frank, a faculty physician at Keck Medicine of USC’s Department of Psychiatry and the Behavioral Sciences; Angel Hsing-Chi Hwang, an assistant professor of communication the USC Annenberg School for Communication and Journalism; as well as their students Yahan Li, Jifan Yao from the Ming Hsieh Department of Electrical and Computer Engineering, John Bosco S. Bunyi of the Suzanne Dworak-Peck School of Social Work.
100 Mental Health Professionals Graded AI Responses on Safety and Quality #
The study is divided into a two-part framework designed to evaluate how LLMs handle mental health support questions.
In the first part of the experiment, the team selected 100 real patient questions from CounselChat, a public forum where licensed therapists respond to anonymous users.
The experiment focused on three widely used LLMs: ChatGPT-4 by OpenAI, Llama 3.3 by Meta and Gemini 1.5 Pro by Google.
The questions covered 20 diverse mental health topics, including depression, trauma and workplace stress. Each question was answered by the AI models, as well as the original top-voted human therapist response from the forum for comparison.
The researchers then recruited 100 mental health professionals, more than 70% of whom were licensed practitioners, to provide 2,000 expert evaluations of 400 responses.
The professionals graded the AI-generated responses across six clinical criteria: overall quality, empathy, specificity, medical advice appropriateness, toxicity and factual consistency.
Overall quality served as a holistic measure of the response, while specificity evaluated how well the answer was tailored to a user’s particular circumstances rather than relying on generic advice. Medical advice appropriateness assessed whether a response included therapeutic or diagnostic guidance that should be provided only by licensed professionals. Toxicity measured the presence of potentially harmful, stigmatizing or dismissive language, while factual consistency examined whether responses aligned with accepted clinical knowledge and avoided unsupported claims.
Liu said the study’s use of open-ended questions sets it apart from many previous evaluations, which often rely on multiple-choice assessments. Because mental health support is highly subjective, evaluating AI responses to real patient questions provides a more realistic measure of how people interact with these systems. The researchers argue that this qualitative approach offers a more effective way to assess AI performance in mental health settings.
After identifying recurring issues in the first phase of the study, the researchers designed a series of stress tests using 120 adversarial questions specifically crafted to trigger known failure patterns. A separate group of experts then reviewed the resulting responses.
The goal of the second phase was to determine whether the identified failure modes were consistently present across the AI models.
In parallel with the human evaluation, the researchers also tested whether LLMs could reliably evaluate themselves. Nine advanced AI models were asked to grade the same responses using the same rubric as the human experts, allowing the team to assess whether AI systems can accurately identify their own shortcomings.
Study Suggests AI Has Promising Potential as a Mental Health Tool #
The study’s results showed that the tested LLMs demonstrated strong performance in general communication, receiving high ratings for overall quality, empathy and specificity from the mental health professionals who evaluated the responses.
Across the board, the AI models were noted for being fluent and supportive, often appearing as helpful as their human counterparts in basic interactions.
Among the models, Llama 3.3 received the highest overall quality ratings, leading in five of the six evaluation dimensions.
The results also showed that ChatGPT-4 was the safest model overall. It was the most likely to include safety disclaimers, with roughly one-third of its responses explicitly declining to answer certain questions and instead recommending consultation with a licensed professional.
Gemini 1.5 Pro was the lowest-performing model overall, but it still received a higher empathy score from evaluators than the online human therapists, when asked to respond to the same questions.
The research also highlighted Gemini’s primary strength as a platform-integrated model, since it powers tools across Google’s ecosystem, including Google Search and Google Assistant.
Overall, the study’s findings suggest that AI could serve as a promising resource for mental health support.
AI as Counselors? Not Quite There Yet, Study Flags Safety Gaps in AI Mental Health Responses #
While the study shows that AI’s promise to become a useful tool in mental health services, safety remains the main concern, as all LLMs struggled with and were flagged for providing varying levels of unauthorized medical advice, raising significant safety concerns. Common issues across all models included overgeneralization, making unsupported assumptions and providing unconstructive feedback.
Gemini 1.5 Pro was most frequently flagged for lacking empathy or emotional attunement (44.1%).
Llama 3.3 was still the most prone to overgeneralization or making judgments without limited context among all models. It was also most frequently flagged for giving unauthorized medical advice.
GPT-4 was frequently flagged for offering unconstructive feedback and showing little personalization or relevance to the patient’s specific situation.
From stress testing the models, AI further showed challenges like unauthorized medication advice, therapy suggestions and symptom speculation. This included recommending specific psychotropic drugs like antidepressants, prescribing techniques like cognitive behavioral therapy (CBT) or mindfulness, or guessing at clinical diagnoses based on limited patient context. These behaviors identified as safety red flags for AI systems operating without professional oversight. Other patterns observed to be unintentionally judgmental, like calling a behavior “not normal,” or giving apathetic responses, when models were stress-tested.
The team also found that when asked to grade their own performance, AI judges were unreliable, as they consistently overestimated their own performance and missed safety risks that human experts easily identified.
Published on July 7th, 2026
Last updated on July 7th, 2026