10:05
2026-05-28
royvanrijn.com
large-language-models
The Anatomy of an LLM
OpenAI's o200k_base tokenizer splits the sentence "If the human brain were so simple that we could understand it, we would be so simple that we couldn't." into 102 tokens, converting text into integer…