{"slug": "show-hn-unigram-encode-bytes-as-words-that-cost-one-llm-token", "title": "Show HN: Unigram: encode bytes as words that cost one LLM token", "summary": "Unigram, a new Rust crate released on crates.io, encodes bytes as words that each cost exactly one LLM token, making an N-byte value cost exactly N tokens under Claude and other models. The bijective codec, available via `cargo add unigram`, offers canonical parsing and tolerant recovery, with benchmarks showing it beats hex, base64url, and base58 at all sizes under Claude, and wins at 4 bytes under GPT-4o's o200k vocabulary.", "body_md": "A bijective codec between bytes and words that cost exactly one LLM token.\n\n`cargo add unigram`\n\n· [crates.io](https://crates.io/crates/unigram) ·\n[docs.rs](https://docs.rs/unigram) · [CHANGELOG](/bleugreen/unigram/blob/main/CHANGELOG.md)\n\n``` php\na14ed61a                          ->  password email share building\n\n8623a771b764ce50bb85371ff65aebe9  ->  links change points high random found\n                                      season events region light const case\n                                      users field table support\n```\n\nAn identifier becomes something you can read. Say it out loud, carry it across a room or between two windows, tell it apart from its neighbour at a glance, recognise it again an hour later — the ordinary things a name affords. Ids spend their lives in prompts, logs, and error messages, being looked at; this makes that free.\n\nOne word is one byte and one token, so a value costs exactly as many tokens as it carries bytes — flat, for every value, with the spaces between words costing nothing. The four words above carry 32 bits in 4 tokens; the sixteen carry 128 in 16.\n\n``` js\nuse unigram::{UnigramId, CheckedUnigramId};\n\nlet id: UnigramId<4> = UnigramId::try_random()?;   // 32 fresh bits, 4 tokens\nprintln!(\"{id}\");                                  // \"password email share building\"\n\nlet returned = UnigramId::<4>::parse(&text)?;      // canonical: exact\nlet salvaged = UnigramId::<4>::recover(&text)?;    // tolerant: forgives a round trip\n\n// One extra word of CRC-8, when a mutated value must not pass as a valid one.\nlet checked: CheckedUnigramId<4> = CheckedUnigramId::try_random()?;\n```\n\nThe bytes are the value; the words are how it is displayed and parsed. Holding it that way means the length is part of the type, equality is byte equality, and there is no question of what format a given value is in — the question a string-shaped API cannot answer and has to guess at.\n\nFree functions (`encode`\n\n, `decode`\n\n, `decode_recovered`\n\n, `try_mint`\n\n) are there for\nvariable-length payloads.\n\n`parse`\n\nis canonical: lowercase alphabet words, single spaces, nothing else. One accepted\nspelling per value, which is what belongs where a value is about to be trusted.\n\n`recover`\n\nforgives what a round trip through a model does — case, separators, line\nwrapping. It reads the whole input, so isolate the candidate first.\n\nBoth refuse an unknown word and name it.\n\nOne word is one byte and one token, so an N-byte value costs exactly N tokens, the same for every value. Mean tokens under Claude, with the worst of 200 deterministic payloads in parentheses:\n\n| encoding | 4 bytes | 8 bytes | 16 bytes | 32 bytes |\n|---|---|---|---|---|\n`unigram` |\n4.0 (4) |\n8.0 (8) |\n16 (16) |\n32 (32) |\n| hex | 6.0 (9) | 11.3 (15) | 21.7 (27) | 42.6 (52) |\n| base64url | 6.3 (9) | 10.8 (14) | 21.3 (25) | 41.2 (48) |\n| base58 | 6.6 (9) | 10.9 (13) | 21.2 (26) | 42.0 (47) |\n\nThe parenthesised figure matters as much as the mean. Every other encoding's cost swings with the value, so a budget built on one has to assume its worst case; this one is known before the value is minted.\n\nHex loses everywhere, at every size, in every family. The GPT vocabularies have memorised\nbase64 fragments, which changes that ranking above 4 bytes — under `o200k`\n\n, base64url\naverages 29.5 tokens for 32 bytes against a flat 32, while `unigram`\n\nstill wins at 4\nbytes (4.0 against 4.5). Nonce and correlation-id widths are what this was built for;\na 32-byte digest is a worse fit, at 224 characters and no token margin left under GPT.\n\nOne token per byte holds space-prefixed **and bare**, so a value costs exactly N at\nthe start of a string, after a space, in JSON, and mid-sentence. The only surcharge is\npunctuation immediately before it. Measured for a 4-byte value against an ideal of 4,\nsweeping **all 256 entries** through the opening and closing positions, worst kept:\n\n| context | GPT-4o | GPT-3.5/4 | GPT-3 | GPT-2 | Llama | Claude |\n|---|---|---|---|---|---|---|\n| start of string | +0 | +0 | +0 | +0 | +0 | +0 |\nin prose, `X.` |\n+0 | +0 | +0 | +0 | +0 | +0 |\nJSON `\"id\":\"X\"` |\n−1 | +0 | +0 | +0 | +0 | +1 |\n| after a newline | +0 | +0 | +0 | +0 | +0 | +1 |\nafter `id: ` |\n−1 | −1 | −1 | −1 | −1 | +0 |\nmarkdown ``X`` |\n+1 | +1 | +1 | +1 | +1 | +0 |\nafter `(` |\n+1 | +1 | +1 | +1 | +1 | +0 |\n\nSo: *one token per byte, plus at most one for punctuation immediately before it* — a\nconstant, never scaling with the payload, and negative where the context ends in a\nspace the value absorbs.\n\nThat is a property of the table, and it was not free. 0.2.0 shipped 22 entries costing\ntwo or three tokens bare, so a value opening with `council`\n\ncost N+2 at the start of a\nstring — and its verifier tested one payload whose opening word happened to be cheap.\nBoth are fixed. The sweep is why the claim needs no exception list.\n\nBIP39, Diceware, the PGP word list, and `what3words`\n\nall predate this and all map\ndata to words. None was chosen for tokenizers, and it shows. BIP39 is the closest\ncomparison — 2048 words, which would be 11 bits each if they were all single tokens:\n\n| wordlist | words | single-token both ways, all families | usable alphabet |\n|---|---|---|---|\n| BIP39 | 2048 | 349 |\n256 → 8 bits/token |\n`unigram` |\n256 | 256 | 256 → 8 bits/token |\n\nOnly 349 of BIP39's 2048 survive the filter, and Claude is the binding constraint at 366. Round 349 down to a power of two and a BIP39-derived encoding lands on exactly 256 entries and exactly 8 bits per token — the same density, from a list that also has no bare-cost or surrounding-context guarantee.\n\nBIP39 optimises for a different thing, and does it well: unique four-character prefixes and human-transcription distance, for seed phrases read off paper. That is worth having. It is not what makes a word cost one token.\n\nTokenizer vocabularies hold their canonical word entries space-prefixed, so the space between two words is absorbed into the word that follows it and costs nothing. No other separator is free. Measured across all five families, an eight-byte value:\n\n| separator | GPT-4o | GPT-3.5/4 | GPT-3 | GPT-2 | Llama | Claude |\n|---|---|---|---|---|---|---|\n| space | 8 | 8 | 8 | 8 | 8 | 8 |\n`_` `.` |\n8 | 8 | 15 | 15 | 15 | 15 |\n`-` |\n11 | 9 | 15 | 15 | 15 | 15 |\n`,` `\\n` |\n13–15 | 12–15 | 15 | 15 | 15 | 15 |\n\nThe join would cost almost as much as the payload. Encoded values travel inside quoted\nstrings in practice, where embedded spaces are free — and `recover`\n\naccepts every one\nof those separators anyway, so a value that comes back joined differently is not lost.\n\n256 entries of lowercase ASCII English, 4 to 10 characters, under five constraints:\n\n**One token, space-prefixed and bare,** under every tokenizer the verifier pins: OpenAI's`r50k_base`\n\n,`p50k_base`\n\n,`cl100k_base`\n\n,`o200k_base`\n\n; the`hf-internal-testing/llama-tokenizer`\n\nSentencePiece artifact at revision`d02ad6cb`\n\n; and`ctok`\n\n1.0.0's`\"5.0\"`\n\ncounter, an offline reconstruction of Claude's tokenizer rather than Anthropic's own — checked against Anthropic's official`count_tokens`\n\nendpoint on`claude-opus-5`\n\nfor all 256 entries, spaced and bare, where it agrees exactly (`verify-claude.py`\n\nreruns it). Those exact artifacts are the claim, not every model that shares a name, and in particular not Llama 3, which tokenizes with tiktoken rather than the SentencePiece model checked here.**No two entries within one character edit, and none a prefix or suffix-derivative of another.** A slipped character, a dropped suffix, or a completed word lands outside the alphabet rather than on a different valid entry.**Nothing charged**— no death, violence, race, gender, religion, or politics. These strings surface unbidden in transcripts, logs, and user-facing errors.**No function words.** A value made of`that`\n\n,`which`\n\n, and`would`\n\nreads as damaged prose rather than as a name.**Frozen.** Byte`n`\n\nis`ALPHABET[n]`\n\n, all 256 slots are occupied, and changing an entry changes what every previously issued value decodes to. A test pins the table's digest. Nothing in an encoded value says which table produced it, so a system that stores these must record`FORMAT_VERSION`\n\nalongside them.\n\nThe crate depends on nothing but the OS CSPRNG, at runtime or under test, and never\ntokenizes. `cargo test`\n\ncovers the codec and the table's structure — sorted, unique,\nlengths, edit distance, prefix and suffix relationships, the frozen digest, and an\nexhaustive sweep of every single-word substitution against the check word. It says\nnothing about cost.\n\nEvery number on this page is printed by `verify-alphabet.py`\n\n, which reads the alphabet\nstraight out of `src/lib.rs`\n\n, re-measures every entry against all five families both\nspace-prefixed and bare, and sweeps all 256 entries through the opening and closing\npositions of every context — with dependencies and the tokenizer revision pinned\nexactly:\n\n```\nuv run verify-alphabet.py\n```\n\nRun it after any edit to the table. A green test suite alone establishes none of what this crate is named for.\n\n`verify-claude.py`\n\nis the audit for the one measurement that is a reconstruction rather\nthan a vocabulary: it re-checks the Claude column against Anthropic's official\n`count_tokens`\n\nendpoint and reports any entry where the two disagree. Needs\n`ANTHROPIC_API_KEY`\n\n; roughly 300 calls with `--bare`\n\n. It last ran clean on every entry.\n\nMIT.", "url": "https://wpnews.pro/news/show-hn-unigram-encode-bytes-as-words-that-cost-one-llm-token", "canonical_source": "https://github.com/bleugreen/unigram", "published_at": "2026-08-18 17:58:17+00:00", "updated_at": "2026-08-18 18:11:17.427564+00:00", "lang": "en", "topics": ["developer-tools", "artificial-intelligence", "large-language-models"], "entities": ["Unigram", "crates.io", "Claude", "GPT-4o", "GPT-3.5", "GPT-3", "GPT-2", "Llama"], "alternates": {"html": "https://wpnews.pro/news/show-hn-unigram-encode-bytes-as-words-that-cost-one-llm-token", "markdown": "https://wpnews.pro/news/show-hn-unigram-encode-bytes-as-words-that-cost-one-llm-token.md", "text": "https://wpnews.pro/news/show-hn-unigram-encode-bytes-as-words-that-cost-one-llm-token.txt", "jsonld": "https://wpnews.pro/news/show-hn-unigram-encode-bytes-as-words-that-cost-one-llm-token.jsonld"}}