04:56
2026-07-16
thegustafson.com
large-language-models
Unicode, Bytes, and What Text Actually Is
Unicode and UTF-8 encoding define how text is represented as bytes, which language models then tokenize into integers. A single visual character can consist of multiple code points and bytes, complica…