Characterizing Narrative Content in Web-scale LLM Pretraining Data

Researchers at the University of Washington and Allen Institute for AI conducted the first fine-grained study of narrative features in Dolma, a 3-trillion-token open LLM pretraining corpus. They developed NarraBERT, a RoBERTa-based model, to analyze narrative structure across 3 million passages, finding that narrative qualities are unequally distributed across pretraining sources and topics. The study highlights gaps in current data curation practices and provides a foundation for understanding how narrative composition affects LLM reasoning tasks.

arXiv:2606.19468v1 Announce Type: new Abstract: The narrative composition of web-scale LLM pretraining corpora remains largely unexplored even though narrative is a fundamental mode of human communication. We present the first fine-grained study of narrative features in Dolma, a 3-trillion-token open pretraining corpus. Drawing on narrative theory, we design a framework spanning three core narrative elements agency, setting, and events operationalized as 11 interpretable dimensions. After sampling and annotating a diverse set of 400 passages, we finetune and validate NarraBERT, a RoBERTa-based model for fine-grained narrative prediction. We apply NarraBERT to 3M passages, resulting in a new dataset, NarraDolma. We find i narrative structure is measurable at scale across extremely heterogeneous data, ii we uncover a continuous, multidimensional narrative structure underlying web text, and iii narrative qualities are unequally distributed across pretraining sources and topics in ways that current curation practices neither measure nor account for. Our framework, dataset, and analyses provide a foundation for understanding how narrative qualities are distributed in LLM pretraining data and for studying how data composition affects narrative reasoning tasks. We publicly release NarraDolma and NarraBERT.