# When Do LLMs Need Human Experts? Evidence for Social Science from Jurisprudential Classification

> Source: <https://aclanthology.org/2026.nlpcss-1.6/>
> Published: 2026-07-13 00:00:00+00:00

[When Do LLMs Need Human Experts? Evidence for Social Science from Jurisprudential Classification](https://aclanthology.org/2026.nlpcss-1.6.pdf)

[Caroline Cheng](/people/caroline-cheng/unverified/),
[Edward Stiglitz](/people/edward-stiglitz/unverified/),
[David Mimno](/people/david-mimno/unverified/),
[Matthew Wilkens](/people/matthew-wilkens/)

##### Abstract

Social scientists increasingly use large language models (LLMs) to classify text at scale, raising a key question: when can LLMs replace expert human annotation? Prior work found that earlier generative models failed on complex social science tasks while fine-tuned BERT succeeded, but whether current frontier-scale models close this gap remained untested. We investigate this question on a challenging legal reasoning task—classifying paragraphs from U.S. Supreme Court opinions as employing formal, grand, or no reasoning. Testing frontier LLMs including GPT-5.2 and leading open-weight alternatives, we find that even the most capable prompted models consistently underperform fine-tuned BERT. Only when high-parameter-count generative LLMs are fine-tuned on human-annotated training data does performance improve, and fine-tuned BERT remains a cost-effective alternative. Contrary to a common view, our results demonstrate that scaling to frontier-size LLMs does not eliminate the need for expert annotation on tasks requiring deep domain expertise—a finding with important implications for computational social science measurement.- Anthology ID:
- 2026.nlpcss-1.6
- Volume:
[Proceedings of the Seventh Workshop on Natural Language Processing and Computational Social Science](/volumes/2026.nlpcss-1/)- Month:
- July
- Year:
- 2026
- Address:
- San Diego
- Editors:
[Dallas Card](/people/dallas-card/),[Anjalie Field](/people/anjalie-field/),[Katherine Keith](/people/katherine-keith/),[Julia Mendelsohn](/people/julia-mendelsohn/)- Venues:
[NLP+CSS](/venues/nlpcss/)|[WS](/venues/ws/)- SIG:
- Publisher:
- Association for Computational Linguistics
- Note:
- Pages:
- 103–112
- Language:
- URL:
[https://aclanthology.org/2026.nlpcss-1.6/](https://aclanthology.org/2026.nlpcss-1.6/)- DOI:
[10.18653/v1/2026.nlpcss-1.6](https://doi.org/10.18653/v1/2026.nlpcss-1.6)- Cite (ACL):
- Caroline Cheng, Edward Stiglitz, David Mimno, and Matthew Wilkens. 2026.
[When Do LLMs Need Human Experts? Evidence for Social Science from Jurisprudential Classification](https://aclanthology.org/2026.nlpcss-1.6/). In*Proceedings of the Seventh Workshop on Natural Language Processing and Computational Social Science*, pages 103–112, San Diego. Association for Computational Linguistics. - Cite (Informal):
[When Do LLMs Need Human Experts? Evidence for Social Science from Jurisprudential Classification](https://aclanthology.org/2026.nlpcss-1.6/)(Cheng et al., NLP+CSS 2026)- PDF:
[https://aclanthology.org/2026.nlpcss-1.6.pdf](https://aclanthology.org/2026.nlpcss-1.6.pdf)