# Build a Tokenizer from Scratch

> Source: <https://github.com/crackr-org/build-a-tokenizer-from-scratch/tree/main>
> Published: 2026-08-30 22:32:46+00:00

Build a byte-level BPE tokenizer from scratch, starting with a simple word tokenizer and working up to a trained multilingual encoding compatible with tiktoken.

Along the way, you’ll implement BPE, optimize training, train your own tokenizer, save it as a .tiktoken model, and ship it in a small web playground.

**Live demo:** [See what you will build by the end of the guide](https://tokenizer-playground.onrender.com/).

This repository contains the complete open-source guide. You can also build the project on [Crackr](https://app.crackr.dev/projects/build-your-own-tokenizer), where each step comes with automated evaluations, progress tracking, and interactive playgrounds.

The full project and all evaluations are free.

[Scale up training](/crackr-org/build-a-tokenizer-from-scratch/blob/main/guide/05-ship-your-tokenizer/01-scale-up-training.md)[Port your tokenizer to tiktoken](/crackr-org/build-a-tokenizer-from-scratch/blob/main/guide/05-ship-your-tokenizer/02-port-your-tokenizer-to-tiktoken.md)[Build a tokenizer playground](/crackr-org/build-a-tokenizer-from-scratch/blob/main/guide/05-ship-your-tokenizer/03-build-a-tokenizer-playground.md)[Where to go from here](/crackr-org/build-a-tokenizer-from-scratch/blob/main/guide/05-ship-your-tokenizer/04-where-to-go-from-here.md)

This guide is available under the [MIT License](/crackr-org/build-a-tokenizer-from-scratch/blob/main/LICENSE).