Model Collapse as Cultural Evolution

A new study published on arXiv demonstrates that model collapse in large language models (LLMs) — the progressive degradation of models trained on their own outputs — follows predictable patterns of cultural evolution. Researchers tested LLaMA-2-7B and Mistral-7B across 10 generations in three languages, finding that compositionality initially rises then falls under unfiltered self-training, a non-monotonic trajectory that matches human behavioral data with 94% accuracy. The findings reframe model collapse as a cultural transmission phenomenon and provide concrete principles for designing self-training pipelines.

arXiv:2605.23054v1 Announce Type: new Abstract: Model collapse, the progressive degradation of LLMs trained on their own outputs, has been characterized statistically but lacks a linguistic explanation for which structures degrade, in what order, and why. We show that iterated learning theory from cultural evolution fills this gap. We derive five falsifiable predictions, distinguish those uniquely discriminative for the theory from confirmatory ones, and test them by self-training LLaMA-2-7B and Mistral-7B over 10 generations in English, German, and Turkish. The critical discriminative finding: compositionality follows a non-monotonic trajectory initially rising, then falling under unfiltered self-training. This signature persists with maximally regular seed data ruling out noise removal and is sustained only by task-grounded filtering, not random filtering, providing the first LLM-scale evidence for the compression-communication tradeoff. All predictions are confirmed with large effect sizes Hedges' $g 1.6$; $\mathrm{BF} {10} 100$ , and LLM regularization gradients closely match human behavioral data $R^2 = 0.94$ . These results reframe model collapse as a cultural transmission phenomenon and yield concrete principles for self-training pipeline design.