{"slug": "crisisbench-an-evaluation-suite-for-crisis-counseling-ai", "title": "CrisisBench: an evaluation suite for crisis counseling AI", "summary": "Researchers released CrisisBench, a privacy-preserving evaluation suite for crisis counseling AI built from 1,085 single-turn tasks drawn from 200 de-identified conversations and 1,185 patient LLMs derived from hotline metadata, with a 23-item LLM judge calibrated against expert counselors. In testing, five GPT models plus draft-then-edit and retrieval-augmented generation all underperformed human counselors on excessive empathy and questioning in single-turn evaluation, while in multi-turn evaluation retrieval-augmented generation scored highest overall (0.961) with all systems within 0.058. The rubric reached human-human Krippendorff's alpha of 0.51 and LLM-human alpha of 0.54 (Pearson r = 0.80), and counselors rated the patient LLMs realistic (86%) and useful (92%); the authors released the rubrics, judge prompts, and evaluation tasks.", "body_md": "arXiv:2611.21622v1 Announce Type: new \nAbstract: Background: Crisis helplines are essential to mental health systems but are often understaffed and underfunded. LLMs could support counselors, but existing benchmarks do not capture the escalation, risk assessment, and de-escalation central to crisis conversations, while privacy restrictions limit use of real transcripts.\nMethods: We present CrisisBench, a privacy-preserving suite for single- and multi-turn crisis counseling evaluation. It includes 1,085 single-turn tasks from 200 de-identified conversations and 1,185 patient LLMs derived from hotline metadata. Patient models and a 23-item LLM judge were calibrated against expert counselors.\nResults: We evaluated five GPT models plus draft-then-edit and retrieval-augmented generation. All systems underperformed human counselors on excessive empathy and questioning in single-turn evaluation. The rubric achieved human-human Krippendorff's $\\alpha$ = 0.51 and LLM-human $\\alpha$ = 0.54 (Pearson r = 0.80). In multi-turn evaluation, retrieval-augmented generation scored highest overall (0.961), with all systems within 0.058. Microskills and style/tone were near ceiling, while productivity and safety/risk accounted for most between-system variation. Counselors rated patient LLMs realistic (86%) and useful (92%).\nConclusion: CrisisBench enables scalable, privacy-preserving evaluation of crisis counseling. Current systems perform strongly on core structural competencies but still diverge from experts on important response-level failure modes. We release the rubrics, judge prompts, and evaluation tasks.", "url": "https://wpnews.pro/news/crisisbench-an-evaluation-suite-for-crisis-counseling-ai", "canonical_source": "https://arxiv.org/abs/2611.21622", "published_at": "2026-11-18 05:00:00+00:00", "updated_at": "2026-11-18 05:17:42.275532+00:00", "lang": "en", "topics": ["ai-safety", "ai-research", "large-language-models", "ai-ethics"], "entities": ["CrisisBench", "GPT", "Krippendorff's alpha"], "also_reported_by": [], "takeaway": "CrisisBench's 23-item LLM judge matched human counselors at Krippendorff's alpha 0.54 (Pearson r = 0.80), and all tested systems, including RAG at 0.961, stayed within 0.058 on multi-turn scoring while still diverging from experts on excessive empathy and questioning.", "recommendation": {"advice": "Calibrate an LLM judge against expert human raters and report agreement (CrisisBench hit alpha 0.54, r = 0.80) before trusting it to score crisis or safety-sensitive responses.", "playbook": "evals-testing", "playbook_url": "https://wpnews.pro/playbooks/evals-testing"}, "alternates": {"html": "https://wpnews.pro/news/crisisbench-an-evaluation-suite-for-crisis-counseling-ai", "markdown": "https://wpnews.pro/news/crisisbench-an-evaluation-suite-for-crisis-counseling-ai.md", "text": "https://wpnews.pro/news/crisisbench-an-evaluation-suite-for-crisis-counseling-ai.txt", "jsonld": "https://wpnews.pro/news/crisisbench-an-evaluation-suite-for-crisis-counseling-ai.jsonld"}}