Tutorial: Build and Train a Byte-Level BPE Tokenizer From Scratch

Hi everyone!

I recently finished building and training a byte-level BPE tokenizer from scratch, and I tried to document the whole process in a step-by-step guide.

The guide covers:

It was trained an 8,192-token vocabulary on 100 MiB of mC4 across eight languages, which means 7,936 learned merges on top of the 256 byte tokens. Training took roughly one hour on a 12-core M2 Pro with 32 GB of memory.

I also tried deployed a live demo where you can try the model.

Guide:

I’d love feedback on the implementation, explanations, benchmarks, or anything you think the guide should cover and it didn’t.