{"slug": "when-do-llms-need-human-experts-evidence-for-social-science-from-jurisprudential", "title": "When Do LLMs Need Human Experts? Evidence for Social Science from Jurisprudential Classification", "summary": "A new study finds that even frontier large language models like GPT-5.2 cannot replace expert human annotation for complex social science tasks, as they underperform fine-tuned BERT when classifying legal reasoning in U.S. Supreme Court opinions. The results challenge the assumption that scaling LLMs eliminates the need for domain expertise, with implications for computational social science measurement.", "body_md": "[When Do LLMs Need Human Experts? Evidence for Social Science from Jurisprudential Classification](https://aclanthology.org/2026.nlpcss-1.6.pdf)\n\n[Caroline Cheng](/people/caroline-cheng/unverified/),\n[Edward Stiglitz](/people/edward-stiglitz/unverified/),\n[David Mimno](/people/david-mimno/unverified/),\n[Matthew Wilkens](/people/matthew-wilkens/)\n\n##### Abstract\n\nSocial scientists increasingly use large language models (LLMs) to classify text at scale, raising a key question: when can LLMs replace expert human annotation? Prior work found that earlier generative models failed on complex social science tasks while fine-tuned BERT succeeded, but whether current frontier-scale models close this gap remained untested. We investigate this question on a challenging legal reasoning task—classifying paragraphs from U.S. Supreme Court opinions as employing formal, grand, or no reasoning. Testing frontier LLMs including GPT-5.2 and leading open-weight alternatives, we find that even the most capable prompted models consistently underperform fine-tuned BERT. Only when high-parameter-count generative LLMs are fine-tuned on human-annotated training data does performance improve, and fine-tuned BERT remains a cost-effective alternative. Contrary to a common view, our results demonstrate that scaling to frontier-size LLMs does not eliminate the need for expert annotation on tasks requiring deep domain expertise—a finding with important implications for computational social science measurement.- Anthology ID:\n- 2026.nlpcss-1.6\n- Volume:\n[Proceedings of the Seventh Workshop on Natural Language Processing and Computational Social Science](/volumes/2026.nlpcss-1/)- Month:\n- July\n- Year:\n- 2026\n- Address:\n- San Diego\n- Editors:\n[Dallas Card](/people/dallas-card/),[Anjalie Field](/people/anjalie-field/),[Katherine Keith](/people/katherine-keith/),[Julia Mendelsohn](/people/julia-mendelsohn/)- Venues:\n[NLP+CSS](/venues/nlpcss/)|[WS](/venues/ws/)- SIG:\n- Publisher:\n- Association for Computational Linguistics\n- Note:\n- Pages:\n- 103–112\n- Language:\n- URL:\n[https://aclanthology.org/2026.nlpcss-1.6/](https://aclanthology.org/2026.nlpcss-1.6/)- DOI:\n[10.18653/v1/2026.nlpcss-1.6](https://doi.org/10.18653/v1/2026.nlpcss-1.6)- Cite (ACL):\n- Caroline Cheng, Edward Stiglitz, David Mimno, and Matthew Wilkens. 2026.\n[When Do LLMs Need Human Experts? Evidence for Social Science from Jurisprudential Classification](https://aclanthology.org/2026.nlpcss-1.6/). In*Proceedings of the Seventh Workshop on Natural Language Processing and Computational Social Science*, pages 103–112, San Diego. Association for Computational Linguistics. - Cite (Informal):\n[When Do LLMs Need Human Experts? Evidence for Social Science from Jurisprudential Classification](https://aclanthology.org/2026.nlpcss-1.6/)(Cheng et al., NLP+CSS 2026)- PDF:\n[https://aclanthology.org/2026.nlpcss-1.6.pdf](https://aclanthology.org/2026.nlpcss-1.6.pdf)", "url": "https://wpnews.pro/news/when-do-llms-need-human-experts-evidence-for-social-science-from-jurisprudential", "canonical_source": "https://aclanthology.org/2026.nlpcss-1.6/", "published_at": "2026-07-13 00:00:00+00:00", "updated_at": "2026-07-13 21:43:37.550785+00:00", "lang": "en", "topics": ["large-language-models", "natural-language-processing", "ai-research"], "entities": ["Caroline Cheng", "Edward Stiglitz", "David Mimno", "Matthew Wilkens", "GPT-5.2", "BERT", "U.S. Supreme Court", "Association for Computational Linguistics"], "alternates": {"html": "https://wpnews.pro/news/when-do-llms-need-human-experts-evidence-for-social-science-from-jurisprudential", "markdown": "https://wpnews.pro/news/when-do-llms-need-human-experts-evidence-for-social-science-from-jurisprudential.md", "text": "https://wpnews.pro/news/when-do-llms-need-human-experts-evidence-for-social-science-from-jurisprudential.txt", "jsonld": "https://wpnews.pro/news/when-do-llms-need-human-experts-evidence-for-social-science-from-jurisprudential.jsonld"}}