A controlled comparison of byte-pair encoding (BPE) and Unigram-LM tokenizers on chemistry SMILES reveals that the two algorithms build near-disjoint subword vocabularies across all 22 tested conditio…
Researchers introduced QuechuaTok, a benchmark evaluating tokenization strategies for Southern Quechua, a low-resource agglutinative language. They found that BPE achieved the lowest fertility rate bu…
A 124M-parameter GPT-2 model trained from scratch on OpenWebText data using a custom deep learning library achieved a validation loss of 2.764 nats and a perplexity of 15.87 after 56,000 steps (27.5B …
A hidden bottleneck in LLM proxy architectures is causing 5-13 millisecond blocking delays per request during tokenization, a CPU-bound operation that most systems treat as instantaneous. In event-loo…