{"slug": "the-lexical-imprint-of-pre-training-data", "title": "The Lexical Imprint of Pre-training Data", "summary": "A new study introduces word-level unigram cross-entropy (UCE), a tokenizer-agnostic, model-free measure of lexical alignment between a pre-training corpus and a benchmark target, to quantify how much zero-shot benchmark accuracy is explained by word-level overlap. The research asks how much of that accuracy is captured by lexical alignment between pre-training data and the benchmark. The work offers a model-free way to measure how much benchmark performance may reflect pre-training data overlap rather than task capability.", "body_md": "How much of zero-shot benchmark accuracy is captured by word-level alignment between a pre-training corpus and the benchmark target? We study this question through word-level unigram cross-entropy (UCE), a tokenizer-agnostic, model-free measure of lexical alignment between a reference corpus and a t", "url": "https://wpnews.pro/news/the-lexical-imprint-of-pre-training-data", "canonical_source": "https://aiflash.com/news/141050/", "published_at": "2026-10-28 10:00:20+00:00", "updated_at": "2026-10-28 10:13:35.118391+00:00", "lang": "en", "topics": ["ai-research", "machine-learning", "large-language-models", "natural-language-processing"], "entities": [], "also_reported_by": [], "alternates": {"html": "https://wpnews.pro/news/the-lexical-imprint-of-pre-training-data", "markdown": "https://wpnews.pro/news/the-lexical-imprint-of-pre-training-data.md", "text": "https://wpnews.pro/news/the-lexical-imprint-of-pre-training-data.txt", "jsonld": "https://wpnews.pro/news/the-lexical-imprint-of-pre-training-data.jsonld"}}