{"slug": "does-morphological-correctness-matter-for-tokenization-a-scrambled-control-study", "title": "Does Morphological Correctness Matter for Tokenization? A Scrambled Control Study in Tagalog", "summary": "A study of Tagalog tokenization found that a morphology-aware tokenizer, MorphAustro-Tok, beat a corpus-matched BPE baseline by 0.84, 1.49 and 3.40 mean sentence-level chrF++ points on machine translation across 33 runs, but a scrambled-morpheme control that randomizes the pseudo-infix moved the gain by less than between-seed dispersion at both controlled fertility levels. MorphAustro-Tok emits the Tagalog root as a contiguous unit followed by a positional infix token, addressing infixes such as s-um-ulat and s-in-ulat that no sequence of BPE merges can recover. The authors report no detectable contribution from morphological correctness at this scale, and neither token count nor model capacity explains the gain.", "body_md": "arXiv:2610.20859v1 Announce Type: new \nAbstract: Typology-aware tokenizers are motivated by the claim that segmentation which respects morpheme boundaries preserves information that frequency-driven segmentation destroys. Studies routinely establish that it helps while assuming the boundaries must be linguistically correct. We separate them. Tagalog marks voice with infixes (s-um-ulat 'wrote', s-in-ulat 'was written'), so the root is not contiguous and no sequence of BPE merges can recover it. We build MorphAustro-Tok, a tokenizer that emits the root as a contiguous unit followed by a positional infix token, and compare it against a corpus-matched BPE baseline and, at two of three fertility levels, against a scrambled-morpheme control that preserves morph count, invertibility and per-type consistency while randomising the pseudo-infix. Across 33 runs, MorphAustro-Tok improves Tagalog machine translation over matched BPE by 0.84, 1.49 and 3.40 mean sentence-level chrF++ points. Token count does not explain the gain, since it grows as segmentation coarsens, and neither does capacity: at the coarsest level the scrambled arm carries more parameters and scores lower. Scrambling the analysis moves the gain by less than between-seed dispersion at both controlled levels, so we detect no contribution from morphological correctness at this scale.", "url": "https://wpnews.pro/news/does-morphological-correctness-matter-for-tokenization-a-scrambled-control-study", "canonical_source": "https://arxiv.org/abs/2610.20859", "published_at": "2026-10-16 04:00:00+00:00", "updated_at": "2026-10-16 05:07:11.918769+00:00", "lang": "en", "topics": ["natural-language-processing", "machine-learning", "ai-research", "large-language-models"], "entities": ["MorphAustro-Tok", "Tagalog", "BPE", "chrF++", "arXiv"], "also_reported_by": [], "alternates": {"html": "https://wpnews.pro/news/does-morphological-correctness-matter-for-tokenization-a-scrambled-control-study", "markdown": "https://wpnews.pro/news/does-morphological-correctness-matter-for-tokenization-a-scrambled-control-study.md", "text": "https://wpnews.pro/news/does-morphological-correctness-matter-for-tokenization-a-scrambled-control-study.txt", "jsonld": "https://wpnews.pro/news/does-morphological-correctness-matter-for-tokenization-a-scrambled-control-study.jsonld"}}