{"slug": "where-to-cut-how-deep-bpe-and-unigram-lm-on-chemistry-smiles", "title": "Where to cut, how deep: BPE and Unigram-LM on chemistry SMILES", "summary": "A controlled comparison of byte-pair encoding (BPE) and Unigram-LM tokenizers on chemistry SMILES reveals that the two algorithms build near-disjoint subword vocabularies across all 22 tested conditions, with cross-algorithm Jaccard overlap never exceeding 0.161. Unigram-LM segments molecules into 29-41% more tokens than BPE, and BPE's segmentation is a strict coarsening of Unigram-LM's on 80-99% of molecules. The findings indicate that subword algorithm choice is a critical modeling decision for chemical language models, not a free default.", "body_md": "arXiv:2607.05691v1 Announce Type: new\nAbstract: Every chemical language model reading SMILES begins with a tokenizer, yet the field has inherited byte-pair encoding (BPE) from natural language with little scrutiny. In natural language, BPE's principal alternative, Unigram-LM, is known to build structurally different vocabularies. Whether that contrast survives in chemistry was open. We report a controlled comparison of BPE and Unigram-LM over a fixed 165-token chemistry base, at the small vocabulary sizes where token embeddings are learnable, across three corpus typologies (diverse, drug-like, natural-products) and both pre-tokenization boundary policies. The two do not converge. In all 22 matched conditions they build near-disjoint subword vocabularies: cross-algorithm Jaccard overlap on the learned pieces never exceeds 0.161, and at most 0.05 once weighted toward the high-frequency pieces a model updates most. Unigram-LM also segments held-out molecules into 29-41% more tokens; the arms largely agree on where to cut but not how deeply, so BPE's segmentation is a strict coarsening of Unigram-LM's on 80-99% of molecules. The separation holds across corpus, boundary, and vocabulary size, persisting even at eight times that scale. The subword algorithm is therefore a modeling decision, not a free default. The study trains no language models.", "url": "https://wpnews.pro/news/where-to-cut-how-deep-bpe-and-unigram-lm-on-chemistry-smiles", "canonical_source": "https://arxiv.org/abs/2607.05691", "published_at": "2026-07-08 04:00:00+00:00", "updated_at": "2026-07-08 04:03:43.075645+00:00", "lang": "en", "topics": ["natural-language-processing", "machine-learning", "large-language-models"], "entities": ["BPE", "Unigram-LM", "SMILES"], "alternates": {"html": "https://wpnews.pro/news/where-to-cut-how-deep-bpe-and-unigram-lm-on-chemistry-smiles", "markdown": "https://wpnews.pro/news/where-to-cut-how-deep-bpe-and-unigram-lm-on-chemistry-smiles.md", "text": "https://wpnews.pro/news/where-to-cut-how-deep-bpe-and-unigram-lm-on-chemistry-smiles.txt", "jsonld": "https://wpnews.pro/news/where-to-cut-how-deep-bpe-and-unigram-lm-on-chemistry-smiles.jsonld"}}