04:56
2026-07-16
thegustafson.com
natural-language-processing
WordPiece, Unigram, and SentencePiece
Three tokenization algorithms—WordPiece, Unigram, and SentencePiece—each split text differently for language models. WordPiece merges pairs to maximize likelihood, Unigram prunes a large initial vocab…