04:00
2026-09-16
arxiv.org
natural-language-processing
The Functionalizer: Lossless Functional Decomposition for Subword Tokenization
A new arXiv paper (2609.15991v1) introduces the Functionalizer, a lossless pre-tokenizer framework that decomposes orthographic and structural word variations into a compositional opcode/operand prefi…