Abstract
Creative text is inherently subjective: any opinion can be equally valid, and there is no objective ground truth. The only way to quantify such subjectivity is to model the crowd of opinions that produces it. Large language model (LLM)-as-a-judge has emerged as a scalable substitute for human raters, but whether an LLM can stand in for the human crowd on AI creativity remains unexplored. We develop a decomposition framework (four main components, 11 sub-components) and a three-step rating procedure, applied identically to human raters and to different LLM judging conditions on AI-generated stories. Humans rate AI creativity with the same subjectivity that creativity research documents for human-authored creative work. Across three structural comparisons (disagreement on the same story including ranking; revision from impressionistic to reflective; sub-component weighting of the final rating), no LLM reproduces all three human patterns. We frame the contribution as a methodological framework plus a quantification of where LLM judges currently fall short of the human ratings in our study. We release all stories, ratings, and code at https://github.com/ku-nlp/creativity-evaluation-artifact.
- Anthology ID:
- 2026.aacl-long.203
- Volume:
- Proceedings of the 15th International Joint Conference on Natural Language Processing and the 5th Conference of the Asia-Pacific Chapter of the Association for Computational Linguistics (Volume 1: Long Papers)
- Month:
- November
- Year:
- 2026
- Address:
- Hengqin, China
- Editors:
- Yue Zhang ,Ruifeng Xu ,Ayu Purwarianti ,Tong Rong ,Jun Suzuki
- Venues:
- [AACL](https://aclanthology.org/venues/aacl/) |[IJCNLP](https://aclanthology.org/venues/ijcnlp/)
- SIG:
- Publisher:
- Association for Computational Linguistics
- Note:
- Pages:
- 4480–4502
- Language:
- URL:
- [https://aclanthology.org/2026.aacl-long.203/](https://aclanthology.org/2026.aacl-long.203/)
- DOI:
- Cite (ACL):
- Pharath Sathya, Yin Jou Huang, and Fei Cheng. 2026. A Framework for Evaluating AI-Generated Creativity: Human–LLM Agreement in Evaluating AI-Generated Stories . InProceedings of the 15th International Joint Conference on Natural Language Processing and the 5th Conference of the Asia-Pacific Chapter of the Association for Computational Linguistics (Volume 1: Long Papers) , pages 4480–4502, Hengqin, China. Association for Computational Linguistics.
- Cite (Informal):
- [A Framework for Evaluating AI-Generated Creativity: Human–LLM Agreement in Evaluating AI-Generated Stories](https://aclanthology.org/2026.aacl-long.203/) (Sathya et al., AACL-IJCNLP 2026)
- PDF:
- [https://aclanthology.org/2026.aacl-long.203.pdf](https://aclanthology.org/2026.aacl-long.203.pdf)