{"slug": "i-built-mlbox-a-browser-workbench-that-shows-how-llm-tokenizers-really-work", "title": "I Built MLBox: A Browser Workbench That Shows How LLM Tokenizers Really Work", "summary": "Mohit Verma, a Class 10 student from Patna, India, built MLBox, an open-source, dependency-free browser workbench that compares seven real LLM tokenizers side by side, along with a neural-net playground, no-GPU project recipes, and an LLM price calculator. The tokenizer engine re-implements both BPE families in roughly 550 lines of JavaScript and matches the upstream Rust implementations token-for-token across 218,877 tokens on all seven models. Verma's demo shows the same Hindi phrase costing 5 tokens under GPT-4o's tokenizer versus 13 under cl100k, illustrating tokenization as a cost and context-window multiplier.", "body_md": "Hi, I'm Mohit Verma, a Class 10 student from Patna, India, learning machine learning by building things. MLBox is my open-source project: a browser workbench for LLM tokenizers, training intuition, and API pricing. Four tools, one static page, zero dependencies — every number is measured in your own browser.\n\nLive demo: [https://mohittt-vermaa.github.io/mlbox/](https://mohittt-vermaa.github.io/mlbox/)\n\nCode: [https://github.com/mohittt-vermaa/mlbox](https://github.com/mohittt-vermaa/mlbox)\n\n**Tokenizer lens** — Paste any text and watch 7 real LLM tokenizers cut it up, token by token: counts, cost per million characters, a language cost matrix, a boundary diff between any two tokenizers, and a shareable link that freezes the whole comparison.\n\n**Micro playground** — A neural net with hand-written backprop training live on canvas. Watch gradient descent solve XOR.\n\n**No-GPU recipes** — 12 curated ML projects that genuinely finish on a laptop or a free Colab session, each with its one gotcha spelled out.\n\n**LLM price bench** — 13 current models from 5 providers, transcribed from each provider's own pricing page with source links, plus a workload calculator and a real-prompt cost estimator.\n\nPaste \"नमस्ते दुनिया\" (\"hello world\" in Hindi, 13 characters) and the tokenizers disagree: GPT-4o's tokenizer needs 5 tokens, Llama 3.1 needs 9, GPT-3.5/4's cl100k needs 13. Same two words, nearly 3x the bill depending on which tokenizer sits in front. Token choice is a cost multiplier and a context-window multiplier, and almost nobody looks at it.\n\nThe tokenizer engine re-implements both BPE families in about 550 lines of dependency-free JavaScript, and the test suite checks it token-for-token against the upstream Rust implementations: 218,877 tokens, exact match on all seven models. There is also an Android app and a Windows desktop build in the releases, and everything runs offline with no API key and no backend — your text never leaves your machine.\n\nI kept reading claims about token costs from blog posts instead of measuring them. MLBox is my way of learning ML properly: build the tool, verify every number, and share it so the next student can measure things themselves.\n\nIf it helps you, a star on GitHub means a lot: [https://github.com/mohittt-vermaa/mlbox](https://github.com/mohittt-vermaa/mlbox)\n\n— Mohit Verma\n\nAll my work: [https://mohittt-vermaa.github.io](https://mohittt-vermaa.github.io)", "url": "https://wpnews.pro/news/i-built-mlbox-a-browser-workbench-that-shows-how-llm-tokenizers-really-work", "canonical_source": "https://dev.to/mohitttvermaa/i-built-mlbox-a-browser-workbench-that-shows-how-llm-tokenizers-really-work-3llc", "published_at": "2026-11-20 18:36:13+00:00", "updated_at": "2026-11-20 18:44:08.639684+00:00", "lang": "en", "topics": ["large-language-models", "ai-tools", "developer-tools", "machine-learning"], "entities": ["Mohit Verma", "MLBox", "GPT-4o", "Llama 3.1", "GitHub", "Colab"], "also_reported_by": [], "takeaway": "The same 13-character Hindi phrase tokenizes to 5 tokens in GPT-4o, 9 in Llama 3.1 and 13 in cl100k, so tokenizer choice can nearly triple both cost and context usage for non-English text.", "recommendation": null, "alternates": {"html": "https://wpnews.pro/news/i-built-mlbox-a-browser-workbench-that-shows-how-llm-tokenizers-really-work", "markdown": "https://wpnews.pro/news/i-built-mlbox-a-browser-workbench-that-shows-how-llm-tokenizers-really-work.md", "text": "https://wpnews.pro/news/i-built-mlbox-a-browser-workbench-that-shows-how-llm-tokenizers-really-work.txt", "jsonld": "https://wpnews.pro/news/i-built-mlbox-a-browser-workbench-that-shows-how-llm-tokenizers-really-work.jsonld"}}