BPE vs. Morphology-Aware Tokenization: A Case Study on Arabic Language Modeling at BabyLM Scale

Abstract

Byte Pair Encoding (BPE) is blind to morphology, a poor fit for Arabic, whose words bundle conjunctions, articles, and inflection into one token. The leading morphology-aware strategy for Arabic, MorphBPE, relies on Farasa, a supervised segmenter disallowed under the BabyLM 2026 rule against off-the-shelf linguistic tools. We study Arabic segmentation under the BabyLM GPT-2 training pipeline and propose Morfessor+MorphBPE, an unsupervised morphology-aware tokenizer. We compare two dimensions: supervised vs. unsupervised morphological segmentation and the merging constraints used by BPE. All tokenizer variants cluster within a narrow accuracy band, with improvements rather on language understanding tasks than knowledge tasks, even though we do not observe consistent improvements over BPE. Furthermore, we find that holding the Farasa segmentation fixed, the MorphBPE constraint sharply lowers perplexity but barely changes downstream accuracy; and unsupervised Morfessor boundaries match supervised Farasa on aggregate while scoring higher on grammaticality (MultiBLiMP). Morfessor+MorphBPE thus offers a practical path to morphology-aware tokenization for Arabic.

- Anthology ID:
- 2026.babylm-main.16
- Volume:
- [Proceedings of the Second BabyLM Workshop](https://aclanthology.org/volumes/2026.babylm-main/)
- Month:

• October
• Year:
• 2026
• Address:
• Budapest, Hungary
• Editors:
• Leshem Choshen ,Ryan Cotterell ,Mustafa Omer Gul ,Jaap Jumelet ,Tal Linzen ,Aaron Mueller ,Suchir Salhan ,Raj Sanjay Shah ,Alex Warstadt ,Ethan Gotlieb Wilcox

- Venues:
- [BabyLM](https://aclanthology.org/venues/babylm/) |[WS](https://aclanthology.org/venues/ws/)
- SIG:
- Publisher:

• Association for Computational Linguistics

- Note:
- Pages:

• 199–208

- Language:
- URL:
- [https://aclanthology.org/2026.babylm-main.16/](https://aclanthology.org/2026.babylm-main.16/)
- DOI:
- Cite (ACL):

• Mohamed Badawy, Marion Di Marco, and Alexander Fraser. 2026. BPE vs. Morphology-Aware Tokenization: A Case Study on Arabic Language Modeling at BabyLM Scale . InProceedings of the Second BabyLM Workshop , pages 199–208, Budapest, Hungary. Association for Computational Linguistics.

- Cite (Informal):
- [BPE vs. Morphology-Aware Tokenization: A Case Study on Arabic Language Modeling at BabyLM Scale](https://aclanthology.org/2026.babylm-main.16/) (Badawy et al., BabyLM 2026)
- PDF:
- [https://aclanthology.org/2026.babylm-main.16.pdf](https://aclanthology.org/2026.babylm-main.16.pdf)