LittleLearner: Language Models Under Pedagogically Controlled Knowledge Exposure

Researchers released LittleCurriculum, an 88B-token pretraining corpus restricted to U.S. elementary school material and explicitly excluding concepts, facts and vocabulary taught above Grade 5, alongside LittleLearner, a 5B-parameter LLM trained from scratch on it. The team reports that post-training and in-context learning let LittleLearner better use existing knowledge but did not raise out-of-scope capabilities, and released both the corpus and model as a developmentally restricted sandbox for studying how models acquire, represent and use data under a defined training scope.

arXiv:2608.13545v3 Announce Type: replace Abstract: Modern language models are trained on heterogeneous web-scale text corpora. Consequently, studying knowledge and skill acquisition is difficult, as prior exposure to related content is hard to characterize. To address this challenge, we introduce LittleCurriculum, a curated 88B-token pretraining corpus tailored to U.S. elementary school material, explicitly excluding concepts, facts, and vocabulary taught above Grade 5. Training a 5B-parameter LLM from scratch on LittleCurriculum yields LittleLearner, a model with sufficient language competence for open-ended evaluation, yet with clear knowledge and capability boundaries mapped to interpretable curriculum guidelines. We release LittleCurriculum and LittleLearner as a developmentally restricted sandbox to study how models acquire, represent, and use data under a well-defined training scope. We illustrate the sandbox's utility in a first suite of experiments on injecting new knowledge through post-training and in-context learning. These methods let LittleLearner better utilize existing knowledge, but do not raise out-of-scope capabilities. Our findings underscore the value of this controlled environment for future investigations.