Member-only story
A language model can’t read. It has no letters and no words — only numbers. So the first thing that happens to your sentence is a translation, and that translation quietly decides your bill. Let’s follow one real sentence all the way through.
Step 1: Break the sentence into tokens
The model works with tokens — fragments usually bigger than a letter and often smaller than a word. Common words stay whole; rare ones split into reusable pieces. Take this sentence:
“I love tokenizing unhappiness.”
It breaks apart roughly like this:
["I", " love", " token", "izing", " un", "happiness", "."]
Seven tokens from four words. Look at the split: love
stayed whole (it's common), but tokenizing
became token
izing
, and unhappiness
became un
happiness
. That izing
piece is reused across organizing, realizing, criticizing — the model learned it once. The lesson is already visible: short common words are cheap, long rare words are expensive.
Step 2: Turn tokens into numbers
Each token has a fixed entry in the model’s vocabulary — an integer ID:
["I", " love", " token", "izing", " un", "happiness", "."]…