Are LLMs becoming similarly creative? Evidence from three years of models

A new arXiv preprint (2608.19437v1) analyzing three years of LLM releases finds a statistically significant decrease in output diversity on open-ended creative tasks, suggesting models are converging in creative substance. The study, which used sentence-embedding similarity on responses to Infinity-Chat100 and the Alternate Uses Task, warns that this homogenization may diminish human agency in human-AI co-creative work.

arXiv:2608.19437v1 Announce Type: new Abstract: Many benchmarks track Large Language Model LLM performance on tasks with verifiable answers, but less is known about how LLM performance is evolving on open-ended tasks, where creativity, originality and diversity may matter as much as quality. As LLMs increasingly support human ideation and creative work, understanding trends in LLM performance on open-ended tasks is critical. This paper presents a preliminary analysis of LLM creative outputs spanning three years of model releases, examining model responses to Infinity-Chat100, a real-world collection of open-ended user queries, and the Alternate Uses Task, an established psychometric creativity assessment. Using sentence-embedding similarity, we examine trends in LLM responses to these prompts. Our findings show a statistically significant decrease in model output diversity over time, suggesting that LLM outputs may be converging in creative substance across models. If this trend persists, LLM-driven homogenization may progressively diminish human agency in human-AI co-creative work, demanding careful consideration of LLMs' role in the human creative process.