Build a byte-level BPE tokenizer from scratch, starting with a simple word tokenizer and working up to a trained multilingual encoding compatible with tiktoken.
Along the way, you’ll implement BPE, optimize training, train your own tokenizer, save it as a .tiktoken model, and ship it in a small web playground.
Live demo: See what you will build by the end of the guide. This repository contains the complete open-source guide. You can also build the project on Crackr, where each step comes with automated evaluations, progress tracking, and interactive playgrounds.
The full project and all evaluations are free.
Scale up trainingPort your tokenizer to tiktokenBuild a tokenizer playgroundWhere to go from here
This guide is available under the MIT License.