One visual "character" can be many code points, and each code point can be many bytes. Watch the three layers come apart.
So. What does a model actually see when you type "Hello"?
A language model doesn't see letters or pixels on a screen. It sees a sequence of numbers. Integers, specifically, pulled from a vocabulary table. And before those integers exist, before any tokenizer touches the text, the raw input is just bytes. A stream of 0x48 0x65 0x6C 0x6C 0x6F
.
That feels manageable for English: five characters, five bytes, five numbers, clean mapping. Then you try café
and the é
at the end is two bytes pretending to be one character. Or 日本語
, where each character is three bytes. Or 👨👩👧👦
, where the single thing you see on screen is twenty-five bytes, seven code points, and one "character."
The word "character" is doing a lot of work in that paragraph. To understand how tokenizers turn text into the integer sequences models consume, we need to get precise about what text actually is. That starts well before machine learning. It starts with encoding.
This is the first entry in Arc 3. We closed out Arc 2 with sequence-to-sequence models and Bahdanau attention. Now we're zooming into the layer that sits before the model: the pipeline that turns raw text into numbers.
Prerequisites: basic comfort with binary and hexadecimal. If you read the floating point post from Arc 1, the bit-manipulation here will feel familiar.
From ASCII to Unicode, the short version #
In the beginning there was ASCII. Seven bits per character, 128 possible values. Uppercase and lowercase Latin letters, digits, punctuation, and a handful of control characters like newline and tab. If you were writing English-language software in 1970, this was all you needed.
The problem showed up the moment anyone tried to use a computer in French, or German, or Japanese. ASCII had no room for accented characters, let alone thousands of CJK ideographs. The response was code pages: region-specific extensions that reused the same byte values (128-255) to mean completely different characters depending on what country you were in. A file encoded as Windows-1252 (Western European) and opened as Windows-1251 (Cyrillic) would render as garbage. If you've ever seen a web page with é
splattered everywhere that should say é
, that's UTF-8 bytes being misread as Latin-1. Same bug family.
Unicode was the fix. The core idea: assign every character in every writing system a unique integer, called a code point. The Latin capital letter A is U+0041. The Japanese character \u65E5
is U+65E5. The pile-of-poo emoji is U+1F4A9. These are abstract identifiers. A code point isn't a byte, and it isn't a glyph. It's an integer in a big lookup table that (as of Unicode 17.0) contains over 159,000 entries spanning 170+ scripts.
This is the first place where the notion of "character" starts to fracture. A code point is not necessarily what a human would call a character. Before we get into that, we need to talk about how these abstract integers are actually stored in memory.
UTF-8: the encoding that won #
A code point is just a number. U+0041 is the integer 65. U+65E5 is the integer 26,085. U+1F4A9 is the integer 128,169. To store these in a file or send them over a network, you need an encoding: a rule for turning code points into bytes.
There are three main Unicode encodings, and the history of which one won tells you a lot about engineering tradeoffs.
UTF-32 is the simplest. Every code point gets exactly 4 bytes. U+0041 becomes 00 00 00 41
. This makes random access trivial (the nth character is at byte offset 4n
) but wastes enormous space for ASCII-heavy text. An English document quadruples in size for no reason.
UTF-16 uses 2 bytes for code points in the Basic Multilingual Plane (U+0000 to U+FFFF) and 4 bytes for everything above that, via a scheme called surrogate pairs. This was a reasonable bet in the early 1990s, when the Unicode Consortium believed 65,536 code points would be enough. It wasn't. And UTF-16 has a nasty property: it's not ASCII-compatible. The byte 0x00
shows up constantly inside normal text, which breaks C string functions and anything that treats null bytes as terminators.
UTF-8 is variable-width: 1 byte for ASCII (U+0000 to U+007F), 2 bytes for Latin extensions and most European scripts (U+0080 to U+07FF), 3 bytes for the rest of the Basic Multilingual Plane including CJK (U+0800 to U+FFFF), and 4 bytes for everything else (U+10000 to U+10FFFF). The encoding scheme uses a leading-bit pattern that makes it self-synchronizing:
| Code point range | Byte 1 | Byte 2 | Byte 3 | Byte 4 |
|---|---|---|---|---|
| U+0000 to U+007F | 0xxxxxxx |
|||
| U+0080 to U+07FF | 110xxxxx |
10xxxxxx |
||
| U+0800 to U+FFFF | 1110xxxx |
10xxxxxx |
10xxxxxx |
|
| U+10000 to U+10FFFF | 11110xxx |
10xxxxxx |
10xxxxxx |
10xxxxxx |
The x
bits carry the actual code point value. The leading bits tell you how many bytes this character occupies. Every continuation byte starts with 10
, so if you land in the middle of a multi-byte sequence, you can scan backwards until you find a byte that does not start with 10
, and that is the start of the current character. That self-synchronization property is why UTF-8 is so robust in practice.
Drop the reader at a random byte. If it's a continuation byte (10xxxxxx), walk left until you hit a start byte. Every byte in UTF-8 tells you, from its top bits alone, whether it's the beginning of a character or the middle of one. No stream-wide scan required.
And because single-byte UTF-8 characters have a leading 0
, every valid ASCII file is automatically a valid UTF-8 file. That backward compatibility is what made adoption possible. You didn't have to re-encode the entire internet, and UTF-8 now accounts for over 98% of web pages.
The easiest way to build intuition for the scheme is to pick a code point and watch its binary bits get slotted into the template:
The bits of the code point (blue) slot into the payload positions of each byte. The prefix bits (gray) are fixed by UTF-8. Click through ASCII, Latin Extended, CJK, and an emoji to see how the number of bytes grows.
A quick math detour: the bit budget #
The encoding scheme is elegant in a precise way. For a 1-byte character, you get 7 payload bits (the leading 0
is overhead). For 2 bytes, you pay 5 bits of overhead (110
10
) and get 11 payload bits, which covers code points. For 3 bytes, you pay 8 bits of overhead and get 16 payload bits (). For 4 bytes, 11 bits of overhead and 21 payload bits (), far more than the 1,114,112 code points Unicode actually allows.
The key insight: the overhead grows slower than the payload. You sacrifice a few leading bits to buy self-synchronization and ASCII compatibility, and the cost is modest. An English document encoded in UTF-8 is byte-for-byte identical to ASCII. A Chinese document is 50% larger than UTF-16 (3 bytes per character vs. 2), but you gain universal compatibility. That was the right tradeoff for the web.
Code points vs. grapheme clusters #
This is where the word "character" falls apart completely.
Consider the letter é. There are two ways to represent it in Unicode:
Precomposed: U+00E9, a single code point, LATIN SMALL LETTER E WITH ACUTE. In UTF-8:0xC3 0xA9
(2 bytes).**Decomposed:** U+0065 (LATIN SMALL LETTER E) followed by U+0301 (COMBINING ACUTE ACCENT). In UTF-8:`0x65 0xCC 0x81`
(3 bytes).
Both render identically. A human sees é
either way. But one is 1 code point and the other is 2. One is 2 bytes and the other is 3. If your code calls len(s)
to determine string length, it will give you a different answer depending on which form was used, and both answers are arguably wrong because the human-visible length is 1.
This gets worse with emoji. The family emoji 👨👩👧👦
is a single visual unit, but it's constructed from four emoji code points glued together with three Zero-Width Joiner (U+200D) characters. That's 7 code points and 25 bytes for what your brain processes as one symbol.
The Unicode Consortium defines the concept of a grapheme cluster to handle this: the unit of text that a human perceives as a single "character." The rules for segmenting text into grapheme clusters are specified in UAX #29 (Unicode Text Segmentation), and they aren't simple. They have to account for combining marks, emoji modifiers, skin tone selectors, regional indicator pairs (flags), and ZWJ sequences.
So we have three different ways to measure the "length" of a string:
Bytes: the raw storage cost. Whatlen(s.encode('utf-8'))
gives you in Python.Code points: the number of Unicode abstract integers. Whatlen(s)
gives you in Python (because Python 3 strings are sequences of code points).Grapheme clusters: what a human would count. What you need a regex library or ICU to compute.
These three numbers agree for basic ASCII text. They diverge wildly for everything else.
Click the examples to see how bytes, code points, and grapheme clusters disagree. The family emoji is 1 grapheme, 7 code points, and 25 bytes.
Worked example #
Let me walk through two specific strings to make this concrete.
The family emoji
The string: 👨👩👧👦
| Layer | Count | Breakdown |
|---|---|---|
| Graphemes | 1 | One visual symbol |
| Code points | 7 | 👨 + ZWJ + 👩 + ZWJ + 👧 + ZWJ + 👦 |
| Bytes (UTF-8) | 25 | 4 + 3 + 4 + 3 + 4 + 3 + 4 | Each emoji person is a code point above U+10000, so they take 4 bytes each in UTF-8. Each ZWJ (U+200D) lives in the BMP and takes 3 bytes. Total: bytes.
In Python 3, len("👨👩👧👦")
returns 7 because Python counts code points. In JavaScript, the same expression returns 11 because JavaScript strings are UTF-16 and each emoji above U+FFFF is a surrogate pair (2 code units each). You get 4 surrogate pairs + 3 single code units = 11. Two languages, two different answers for the "length" of the same visual symbol, and neither is the "1" that a human would give.
Vietnamese with combining diacritics
The Vietnamese word Việt
("Vietnamese") in NFD (fully decomposed) form:
V
-
i -
e -
combining circumflex + combining dot below +
t
| Layer | Count | Breakdown |
|---|---|---|
| Graphemes | 4 | V, i, ệ, t |
| Code points | 6 | V, i, e, combining circumflex, combining dot below, t |
| Bytes (UTF-8) | 8 | 1 + 1 + 1 + 2 + 2 + 1 |
In NFC form, the same word is V
-
i -
ệ
(precomposed e with circumflex and dot below) + t
, which is 4 code points and 7 bytes. Same word, same visual rendering, different internal representation.
This is what Unicode normalization is for. NFC (Canonical Decomposition, followed by Canonical Composition) collapses combining sequences into precomposed code points where one exists. NFD (Canonical Decomposition) expands precomposed characters into their base + combining mark sequences. If you want string comparison to work reliably, you normalize first. Most tokenizers do exactly this as a preprocessing step.
Implementation sketch: UTF-8 encoding from scratch #
Let me write a bare-bones UTF-8 encoder and decoder in Python to make the bit-manipulation tangible. You would never do this in production (Python's built-in .encode('utf-8')
is faster and correct), but building it once makes the encoding scheme stick.
The bit operations map directly to the table above. For a 2-byte encoding, we shift the code point right by 6 to get the high bits, OR them with the 110
prefix, then mask the low 6 bits and OR with the 10
prefix. Each additional byte adds one more layer of shift-and-mask.
The decoder is the mirror image. You inspect the leading bits of the first byte to decide how many bytes to consume, then mask away the prefix bits and reassemble the payload. The 10xxxxxx
check on continuation bytes is how you detect corruption: if a continuation byte doesn't start with 10
, the stream is malformed.
Normalization: when identical text has different bytes #
I keep circling back to this because it matters so much for tokenization. Two Python strings can render identically on screen, pass every visual inspection, and yet be completely different byte sequences.
This isn't an edge case. Web forms, copy-pasted text, and files from different operating systems routinely contain mixed normalization. macOS, for historical reasons, tends to produce NFD-normalized filenames. If a tokenizer doesn't normalize its input, the same word can tokenize differently depending on how the user typed it. Most serious tokenizers (SentencePiece, the HuggingFace tokenizers library) apply NFKC normalization as a first step.
Why this matters for language models #
So where does this leave us with models?
A language model doesn't see characters, code points, or grapheme clusters. It sees tokens: integers from a fixed vocabulary. The tokenizer is the bridge between raw text and those integers, and it has to make a fundamental choice about its input representation.
The original BPE algorithm (as used in early NMT systems) operated on Unicode characters. You start with a vocabulary of individual characters and iteratively merge the most frequent pairs. This mostly works, but it has a problem: the initial vocabulary has to include every Unicode character that might appear in the training data. For multilingual models, that initial vocabulary can be enormous, and rare characters (from scripts with low representation in the training data) consume vocabulary slots that could be used for more useful subword units.
GPT-2 made a different choice. Radford et al. ran BPE on raw UTF-8 bytes. The initial vocabulary is just 256 entries: one per possible byte value. Every piece of text in every language can be represented as a sequence of these 256 bytes. The tokenizer is completely language-agnostic at the byte level. It doesn't need special handling for Chinese, Arabic, or Devanagari, because they're all just byte sequences.
The tradeoff is sequence length. "Hello" is 5 bytes and might tokenize into 1-2 tokens. 日本語
is 9 bytes and tokenizes into more. The family emoji is 25 bytes. Byte-level tokenization means non-Latin scripts and emoji produce longer token sequences, so the model spends more of its context window (and more compute) processing them. Speakers of languages with complex scripts pay a real computational tax.
Japanese “hello” costs 3x the bytes of English “hello.” Byte-level tokenizers inherit this ratio.
The same greeting in five scripts. UTF-8's variable-width encoding means non-Latin text produces more bytes, and byte-level tokenizers pass that cost straight through to sequence length.
This is the tension that runs through all of Arc 3. Every tokenization strategy is a tradeoff between vocabulary size, sequence length, and language coverage. We'll dig into the mechanics in the next post on Byte-Pair Encoding, where we'll implement BPE from scratch and watch it learn to merge byte sequences into subwords.
Misconceptions #
"Unicode is an encoding." Unicode is a character set: a mapping from abstract characters to code point integers. UTF-8, UTF-16, and UTF-32 are encodings of that character set. Conflating the two is the source of most encoding confusion.
"Python len() tells me how many characters are in a string." It tells you how many code points are in a string. For text with combining characters, emoji sequences, or mixed normalization, the number of code points can be very different from the number of things a reader would call characters. If you need human-perceived length, you need grapheme cluster segmentation.
"Emoji are 4 bytes." A single emoji code point above U+FFFF is 4 bytes in UTF-8. But many emoji are multi-code-point sequences (flags, skin tones, ZWJ families) that can be 8, 15, or 25+ bytes. The byte count depends on which emoji.
What's next #
We covered the full stack from ASCII through UTF-8 to grapheme clusters, and saw why the encoding layer creates an uneven playing field for multilingual models.
The next post covers ** Byte-Pair Encoding from Scratch**, where we build a tokenizer from nothing and watch it learn to merge byte sequences into subwords.
Additional reading (and watching) #
- W3Techs. "Usage Statistics of Character Encodings for Websites." As of 2026, UTF-8 is used by 98.3% of all websites. w3techs.com/technologies/overview/character_encoding - Unicode Consortium. "UAX #29: Unicode Text Segmentation." Defines grapheme cluster boundaries, word boundaries, and sentence boundaries. unicode.org/reports/tr29/ - Kudo, T. & Richardson, J. (2018). SentencePiece: A simple and language independent subword tokenizer and detokenizer for Neural Text Processing. EMNLP 2018. - Radford, A., et al. (2019). "Language Models are Unsupervised Multitask Learners." OpenAI. Introduced byte-level BPE to avoid the need for Unicode-aware preprocessing.
- Petrov, A., et al. (2024). Language Model Tokenizers Introduce Unfairness Between Languages. NeurIPS 2024. Quantifies the token fertility gap across 17,000+ languages. - Limisiewicz, T., et al. (2025). "Grapheme Pair Encoding: A Subword Tokenizer for Improved Morphological and Multilingual Coverage." Uses grapheme clusters as BPE atoms instead of bytes.
- Pagnoni, A., et al. (2024). Byte Latent Transformer: Patches Scale Better Than Tokens. Meta FAIR. Introduces dynamic byte patching as an alternative to fixed tokenization.