A Framework for Evaluating AI-Generated Creativity: Human–LLM Agreement in Evaluating AI-Generated Stories

A study by Pharath Sathya, Yin Jou Huang and Fei Cheng, published in the Proceedings of AACL-IJCNLP 2026 (pages 4480–4502), found that no large language model judge reproduced all three human rating patterns when evaluating AI-generated stories. The authors built a decomposition framework with four main components and 11 sub-components plus a three-step rating procedure, applied identically to human raters and several LLM judging conditions, and tested three structural comparisons: disagreement on the same story including ranking, revision from impressionistic to reflective judgment, and sub-component weighting of the final rating. The team released all stories, ratings and code at github.com/ku-nlp/creativity-evaluation-artifact, framing the work as a methodological framework plus a quantification of where LLM judges fall short of human ratings.

Abstract Creative text is inherently subjective: any opinion can be equally valid, and there is no objective ground truth. The only way to quantify such subjectivity is to model the crowd of opinions that produces it. Large language model LLM -as-a-judge has emerged as a scalable substitute for human raters, but whether an LLM can stand in for the human crowd on AI creativity remains unexplored. We develop a decomposition framework four main components, 11 sub-components and a three-step rating procedure, applied identically to human raters and to different LLM judging conditions on AI-generated stories. Humans rate AI creativity with the same subjectivity that creativity research documents for human-authored creative work. Across three structural comparisons disagreement on the same story including ranking; revision from impressionistic to reflective; sub-component weighting of the final rating , no LLM reproduces all three human patterns. We frame the contribution as a methodological framework plus a quantification of where LLM judges currently fall short of the human ratings in our study. We release all stories, ratings, and code at https://github.com/ku-nlp/creativity-evaluation-artifact. - Anthology ID: - 2026.aacl-long.203 - Volume: - Proceedings of the 15th International Joint Conference on Natural Language Processing and the 5th Conference of the Asia-Pacific Chapter of the Association for Computational Linguistics Volume 1: Long Papers https://aclanthology.org/volumes/2026.aacl-long/ - Month: - November - Year: - 2026 - Address: - Hengqin, China - Editors: - Yue Zhang https://aclanthology.org/people/yue-zhang/unverified/ , Ruifeng Xu https://aclanthology.org/people/ruifeng-xu/ , Ayu Purwarianti https://aclanthology.org/people/ayu-purwarianti/ , Tong Rong https://aclanthology.org/people/tong-rong/unverified/ , Jun Suzuki https://aclanthology.org/people/jun-suzuki/ - Venues: - AACL https://aclanthology.org/venues/aacl/ | IJCNLP https://aclanthology.org/venues/ijcnlp/ - SIG: - Publisher: - Association for Computational Linguistics - Note: - Pages: - 4480–4502 - Language: - URL: - https://aclanthology.org/2026.aacl-long.203/ https://aclanthology.org/2026.aacl-long.203/ - DOI: - Cite ACL : - Pharath Sathya, Yin Jou Huang, and Fei Cheng. 2026. A Framework for Evaluating AI-Generated Creativity: Human–LLM Agreement in Evaluating AI-Generated Stories https://aclanthology.org/2026.aacl-long.203/ . In Proceedings of the 15th International Joint Conference on Natural Language Processing and the 5th Conference of the Asia-Pacific Chapter of the Association for Computational Linguistics Volume 1: Long Papers , pages 4480–4502, Hengqin, China. Association for Computational Linguistics. - Cite Informal : - A Framework for Evaluating AI-Generated Creativity: Human–LLM Agreement in Evaluating AI-Generated Stories https://aclanthology.org/2026.aacl-long.203/ Sathya et al., AACL-IJCNLP 2026 - PDF: - https://aclanthology.org/2026.aacl-long.203.pdf https://aclanthology.org/2026.aacl-long.203.pdf