The Lexical Imprint of Pre-training Data

A new study introduces word-level unigram cross-entropy (UCE), a tokenizer-agnostic, model-free measure of lexical alignment between a pre-training corpus and a benchmark target, to quantify how much zero-shot benchmark accuracy is explained by word-level overlap. The research asks how much of that accuracy is captured by lexical alignment between pre-training data and the benchmark. The work offers a model-free way to measure how much benchmark performance may reflect pre-training data overlap rather than task capability.

How much of zero-shot benchmark accuracy is captured by word-level alignment between a pre-training corpus and the benchmark target? We study this question through word-level unigram cross-entropy UCE , a tokenizer-agnostic, model-free measure of lexical alignment between a reference corpus and a t