Where to cut, how deep: BPE and Unigram-LM on chemistry SMILES
A controlled comparison of byte-pair encoding (BPE) and Unigram-LM tokenizers on chemistry SMILES reveals that the two algorithms build near-disjoint subword vocabularies across all 22 tested conditions, with cross-algor…