A new study introduces word-level unigram cross-entropy (UCE), a tokenizer-agnostic, model-free measure of lexical alignment between a pre-training corpus and a benchmark target, to quantify how much zero-shot benchmark accuracy is explained by word-level overlap. The research asks how much of that accuracy is captured by lexical alignment between pre-training data and the benchmark. The work offers a model-free way to measure how much benchmark performance may reflect pre-training data overlap rather than task capability.
How much of zero-shot benchmark accuracy is captured by word-level alignment between a pre-training corpus and the benchmark target? We study this question through word-level unigram cross-entropy (UCE), a tokenizer-agnostic, model-free measure of lexical alignment between a reference corpus and a t
EU-based hosting, git-push deploys, automatic HTTPS, no cold starts. Free tier with a custom domain — perfect for shipping the agent you just read about.