BPE vs. Morphology-Aware Tokenization: A Case Study on Arabic Language Modeling at BabyLM Scale

A BabyLM 2026 workshop paper by Mohamed Badawy, Marion Di Marco and Alexander Fraser found no consistent downstream accuracy gains from morphology-aware tokenization over standard Byte Pair Encoding for Arabic language modeling at small scale. The authors propose Morfessor+MorphBPE, an unsupervised tokenizer that avoids the supervised Farasa segmenter barred by BabyLM 2026 rules against off-the-shelf linguistic tools; unsupervised Morfessor boundaries matched supervised Farasa on aggregate and scored higher on grammaticality (MultiBLiMP). Holding Farasa segmentation fixed, the MorphBPE constraint sharply lowered perplexity but barely changed downstream accuracy, and all tokenizer variants clustered within a narrow accuracy band, with gains appearing on language understanding rather than knowledge tasks.

Abstract Byte Pair Encoding BPE is blind to morphology, a poor fit for Arabic, whose words bundle conjunctions, articles, and inflection into one token. The leading morphology-aware strategy for Arabic, MorphBPE, relies on Farasa, a supervised segmenter disallowed under the BabyLM 2026 rule against off-the-shelf linguistic tools. We study Arabic segmentation under the BabyLM GPT-2 training pipeline and propose Morfessor+MorphBPE, an unsupervised morphology-aware tokenizer. We compare two dimensions: supervised vs. unsupervised morphological segmentation and the merging constraints used by BPE. All tokenizer variants cluster within a narrow accuracy band, with improvements rather on language understanding tasks than knowledge tasks, even though we do not observe consistent improvements over BPE. Furthermore, we find that holding the Farasa segmentation fixed, the MorphBPE constraint sharply lowers perplexity but barely changes downstream accuracy; and unsupervised Morfessor boundaries match supervised Farasa on aggregate while scoring higher on grammaticality MultiBLiMP . Morfessor+MorphBPE thus offers a practical path to morphology-aware tokenization for Arabic. - Anthology ID: - 2026.babylm-main.16 - Volume: - Proceedings of the Second BabyLM Workshop https://aclanthology.org/volumes/2026.babylm-main/ - Month: - October - Year: - 2026 - Address: - Budapest, Hungary - Editors: - Leshem Choshen https://aclanthology.org/people/leshem-choshen/ , Ryan Cotterell https://aclanthology.org/people/ryan-cotterell/ , Mustafa Omer Gul https://aclanthology.org/people/mustafa-omer-gul/ , Jaap Jumelet https://aclanthology.org/people/jaap-jumelet/unverified/ , Tal Linzen https://aclanthology.org/people/tal-linzen/ , Aaron Mueller https://aclanthology.org/people/aaron-mueller/ , Suchir Salhan https://aclanthology.org/people/suchir-salhan/ , Raj Sanjay Shah https://aclanthology.org/people/raj-sanjay-shah/ , Alex Warstadt https://aclanthology.org/people/alex-warstadt/ , Ethan Gotlieb Wilcox https://aclanthology.org/people/ethan-wilcox/ - Venues: - BabyLM https://aclanthology.org/venues/babylm/ | WS https://aclanthology.org/venues/ws/ - SIG: - Publisher: - Association for Computational Linguistics - Note: - Pages: - 199–208 - Language: - URL: - https://aclanthology.org/2026.babylm-main.16/ https://aclanthology.org/2026.babylm-main.16/ - DOI: - Cite ACL : - Mohamed Badawy, Marion Di Marco, and Alexander Fraser. 2026. BPE vs. Morphology-Aware Tokenization: A Case Study on Arabic Language Modeling at BabyLM Scale https://aclanthology.org/2026.babylm-main.16/ . In Proceedings of the Second BabyLM Workshop , pages 199–208, Budapest, Hungary. Association for Computational Linguistics. - Cite Informal : - BPE vs. Morphology-Aware Tokenization: A Case Study on Arabic Language Modeling at BabyLM Scale https://aclanthology.org/2026.babylm-main.16/ Badawy et al., BabyLM 2026 - PDF: - https://aclanthology.org/2026.babylm-main.16.pdf https://aclanthology.org/2026.babylm-main.16.pdf