04:56
2026-07-16
thegustafson.com
natural-language-processing
WordPiece, Unigram, and SentencePiece
WordPiece, Unigram, and SentencePiece are three subword tokenization algorithms that differ in how they split text, with WordPiece using a likelihood-maximizing merge criterion and a ## continuation m…