How an LLM Turns Words Into Numbers — and What Those Numbers Cost

A language model cannot read letters or words, only numbers, so every input sentence is first broken into tokens — fragments that are usually larger than a letter and often smaller than a word — and each token is assigned a fixed integer ID from the model's vocabulary. Common words stay whole and are cheap, while long rare words split into reusable pieces that drive up the token count and therefore the cost, as illustrated by the sentence "I love tokenizing unhappiness" which becomes seven tokens from four words.

Member-only story How an LLM Turns Words Into Numbers — and What Those Numbers Cost A language model can’t read. It has no letters and no words — only numbers. So the first thing that happens to your sentence is a translation, and that translation quietly decides your bill. Let’s follow one real sentence all the way through. Step 1: Break the sentence into tokens The model works with tokens — fragments usually bigger than a letter and often smaller than a word. Common words stay whole; rare ones split into reusable pieces. Take this sentence: “I love tokenizing unhappiness.” It breaks apart roughly like this: "I", " love", " token", "izing", " un", "happiness", "." Seven tokens from four words. Look at the split: love stayed whole it's common , but tokenizing became token + izing , and unhappiness became un + happiness . That izing piece is reused across organizing , realizing , criticizing — the model learned it once. The lesson is already visible: short common words are cheap, long rare words are expensive. Step 2: Turn tokens into numbers Each token has a fixed entry in the model’s vocabulary — an integer ID: "I", " love", " token", "izing", " un", "happiness", "." …