#
This is a Claude Opus 5.0 summary of the bytelex week-long failures leading to a single success that cascaded into a series of successful prototypes.
A show-and-tell about byte-level models, tokenizer-free structure, and why we think bytes are the right shared substrate for models that need to talk to each other.
#
- We train small byte-level language models — no tokenizer, 256 symbols, that’s the whole alphabet.
- Separately we maintain a library that extracts vocabularies from other people’s tokenizers and studies them as pure structure (no corpus, no weights, no model).
- Those two things met this week. We turned 11 tokenizer vocabularies into a single fixed coordinate space over 3-byte windows, with no corpus statistics anywhere, and asked it to predict which invented words our trained byte model would get wrong.
- Spearman correlation with the model’s measured per-word accuracy: −0.98 (n=8 words). A corpus-derived statistic on the same words scored −0.62.
- Then we used the map to generate training data aimed at the predicted weak spots. The failure pattern flattened, and the fix transferred to words that were never in any training set: held-out accuracy went from**.53 / .40** (two seeds) to**.84 / .79** .
- The map costs ~200 ns per lookup and occupies 0.58% of its own address space. It builds in seconds.
Links at the bottom. Everything below is measured; where a number is shaky I say so.
#
The model. A 237M-parameter byte-level language model trained on about 16B tokens of mostly-English text. It reads raw UTF-8. Nothing is tokenized. One detail matters for later: its input embedding for position t is the sum of three lookups — the current byte, the previous byte, and the one before that. So the thing the model actually “sees” at each step is, structurally, a 3-byte window. Hold that thought.
The adapters. We don’t retrain the model to teach it a behavior. We attach a small module (about 8.5M parameters, ~4,000 training steps, roughly two hours on a single RTX 4090, ~4 GB of VRAM) and train only that, with the base model frozen. It detaches cleanly — with the adapter switched off, the model’s outputs are bit-identical to the untouched original. Cheap experiments, no risk to the base weights.
The library. A separate, model-free thing: it pulls the vocabulary out of any tokenizer (we’ve published extractions for twelve: GPT-2, cl100k, o200k, Qwen, DeepSeek-V3, Llama-3.1, Mistral, SmolLM2, T5, XLM-R, BERT, ByT5) and analyzes the byte structure of the tokens themselves. One early result from it: somewhere between 27% and 36% of every BPE vocabulary we looked at consists of tokens that are internally divided — multiple byte-units wearing one token ID. Run the same analysis on ByT5, where each token is literally one byte, and you get exactly zero internal structure, which is the control working correctly.
#
We wanted to know whether a small adapter could learn to actually chain reasoning steps rather than pattern-match. The task:
If someone is harl, then they are blim.
If someone is blim, then they are quen.
If someone is quen, then they are torv.
If someone is torv, then they are vell.
If someone is vell, then they are sook.
Wren is harl. What follows?
Reply with the final answer only: sook. The words are invented, so there’s no world knowledge to lean on — only the chain. Five steps. Chance among the listed words is about 1/6.
First attempt, with a purpose-built recurrent module bolted on: 63%. Looked great. Then we shuffled the order of the rules — which changes nothing logically — and it fell to 17%, i.e. chance. The model had learned that on our nicely-ordered prompts, the answer is always the consequent of the last rule listed. Worse, a size-matched non-recurrent control scored 87% on the same shortcut. Our fancy module was a worse pattern-matcher than a plain one.
The honest read: the task was broken, not the architecture. So we changed one thing — every training example now renders its five rules in a fresh random order — and threw the custom module away, using the ordinary adapter.
| setup | in-order | shuffled | in-vocabulary check | | bolted-on recurrent module, ordered data | .633 | .167 (chance) | .993 | | its size-matched control, ordered data | .867 | .207 (chance) | 1.000 | | plain adapter, shuffle-blocked data (seed 1) | .527 | .553 | 1.000 | | plain adapter, shuffle-blocked data (seed 2) | .400 | .493 | .980 |
The shuffled column is the one that matters. It’s now as good as or better than the in-order column, meaning the ordering shortcut is gone and something order-invariant is doing the work. And this was measured on invented words the adapter had never trained on — the training vocabulary and the exam vocabulary were disjoint. No scaffold, no “let’s think step by step”, no visible chain — the answer comes out directly.
Lesson we keep relearning: when a model looks smart, try scrambling something that shouldn’t matter.
#
Roughly half the answers were still wrong, so we took every miss apart.
The result surprised us. 73% of wrong answers were correctly-spelled words from the prompt’s own list — just the wrong one. The other 27% were the right word overrun into English (“prin” → “print”, 15 times out of prin’s 18 misses). And one word swallowed everything: 63% of all wrong answers were the single word “sook.”
Then the teacher-forced loss, byte by byte:
| position | loss on the correct answer | | first byte, items the model got right | 0.06 | | first byte, items the model got wrong | 4.41 | | every byte after the first, all items | exactly 0.00 |
The entire decision is the first byte. Once the first letter is committed, the rest of the word is free. That reframed the problem completely: it isn’t a reasoning failure and it isn’t a spelling failure, it’s the English byte-prior hijacking the answer at the moment of commitment, whenever the chain computation isn’t confident.
And it was strongly word-specific:
| invented word | model accuracy | | harl | 18/18 (100%) | | sook | 14/15 (93%) | | vell | 15/20 (75%) | | blim | 14/24 (58%) | | mund | 9/19 (47%) | | quen | 6/21 (29%) | | torv | 3/16 (19%) | | prin | 0/18 (0%) |
Which raises the obvious question: could we have predicted that ladder in advance?
#
Here’s the idea, and it’s simple enough to be suspicious of.
Take the 3-byte window as an address. There are 256³ = 16,777,216 possible addresses — a complete, fixed, deterministic space that exists before any data does. Any byte string is a path through overlapping addresses. sook is two steps: soo → ook.
Now paint weight onto that space using nothing but tokenizer vocabularies. No corpus. No text sampling. No model weights. BPE vocabularies are merge-ordered, so a token’s ID is already a frequency proxy — that’s the only numeric input we use, and in the strongest variant we don’t even use that, just occupancy summed across tokenizers.
For each address you can then compute small things: the entropy over what byte comes next, the share taken by the single most dominant continuation, and whether word-like units are ever observed ending there. That last one is the interesting one — call it “can a unit plausibly stop here?”
The prediction: a made-up word is hard for a byte model precisely when English refuses to let it end. torv and quen and prin sit at addresses where one continuation dominates completely — the byte prior wants to keep going (print, printer), so the model overruns or bails to a safer word.
Test against the measured ladder above:
| where the weights came from | correlation with model accuracy | did it name the worst three? | | corpus statistics (8.2 MB of English text) | −0.62 | yes | | one tokenizer, merge-rank weighted | −0.79 | yes | | all 11 tokenizers, occupancy consensus | −0.98 | yes |
Every variant we tried — five of them, including two crude ones — identified the same worst three words, exactly. Two different per-address statistics (dominance and “can it end here”) independently picked out the same trio.
Fairness notes, because this is the part most likely to be oversold: n = 8 words. A Spearman of −0.98 over eight points is a suggestive number, not a law. The rank-exactness of the worst three across five independent weighting schemes is the more robust part of the signal. And the real test isn’t correlation at all — it’s whether acting on the prediction changes anything.
#
If the map knows which words are hard, it can mint them. So we had it generate a training vocabulary: 48 invented words, half deliberately easy to end, half deliberately hard, screened against the word surfaces of all 11 tokenizers (273,689 of them) so nothing real slipped in, and screened against each other so no two words shared structure. Then we retrained the adapter with the same recipe and everything else identical. Difficulty as a dial, not an accident.
The exam was the old word list — never trained on in either run:
| training vocabulary | held-out accuracy, seed 1 / seed 2 | failure-ladder correlation | | 12 hand-picked invented words | .527 / .400 | −0.98 | | 48 map-generated (24 easy / 24 hard) | .840 / .793 | +0.05 / −0.21 |
Two things happened at once, on both seeds. Accuracy on words the model had never seen rose by about 35 points. And the failure ladder flattened to noise — the thing the map predicted so well became unpredictable, because it had stopped happening. The weakness was trainable, and training it away on one set of hard words transferred to a different set of hard words.
That’s the result we’re actually excited about. Not “we got a higher number,” but: a statistic computed from tokenizer vocabularies, with no corpus and no model, identified a behavioral weakness in a trained network precisely enough to design the cure.
It also found the next problem, which is what a good instrument does. In the new exam we’d deliberately included pairs of words sharing a two-letter opening. When the answer’s look-alike partner is elsewhere in the same prompt, accuracy is .26; when it isn’t, .80. Same 22/85 split on both seeds. So the model commits at byte one and then fails to discriminate at bytes two and three. That’s the current work.
One honest wart: the harder training data made the adapter noisier on unrelated text — about three times the drift of the previous run, past the threshold we hold ourselves to. There’s a knob for that (a loss term that penalizes the adapter for changing the base model’s behavior on text that isn’t its job), and the fix run is on the card right now. First seed came back clean — drift back under the bar and accuracy slightly up — but one seed is one seed, so treat that as in-flight, not a result.
#
Here’s the larger reason we care, beyond invented words.
Every text model consumes bytes eventually. A tokenizer is just a chopping strategy laid over the same underlying stream. Models with different tokenizers can’t easily exchange structure — you can’t line up their vocabularies, you can’t say what one knows that the other has never encountered, and when you try to pass supervision between them, some of it silently evaporates at the boundaries.
But if you express every vocabulary as a field over one fixed address space, those questions become arithmetic:
| | occupied addresses | of which Chinese-lead | | Qwen’s vocabulary | 59,372 | 8,574 | | an English-centric pair | 12,798 | 106 | | shared between them | 10,973 | — | | in Qwen, absent from the English side | 48,399 | — | | in the English side, absent from Qwen | 1,825 | — |
That “48,399” is a number you can compute in seconds, before training anything: it is exactly the region a Chinese-capable teacher inhabits that an English-trained byte student has never witnessed. And it’s not an approximation — CJK characters encode as exactly three UTF-8 bytes, so at character boundaries one Chinese character is precisely one address. We verified that a specific two-character string resolves to two live addresses in Qwen’s field.
The practical upshot: when a teacher model supervises a byte-level student across a tokenizer mismatch, the portion of its signal landing in never-witnessed territory stops being invisible loss and becomes a measured, reportable quantity — something you can pad for, disclose, or build curriculum against.
And the space is cheap enough to actually use. The 11-tokenizer consensus occupies 96,563 addresses — 0.58% of the 16.7M. Everything else is empty and costs nothing: absence is just a missing dictionary key, so a lookup on empty space returns zero and takes no memory. Lookups run at ~200 ns (100,000 of them in 0.02 s). A full field builds from a vocabulary file in 1–12 seconds.
The last piece is the one that made us take this seriously rather than treating it as bookkeeping. Remember the model’s input embedding: current byte plus the two before it. That is a 3-byte window. The map’s coordinate system and the model’s input composition are the same space, arrived at independently — one from architecture, one from vocabulary analysis. So a prediction made in map coordinates is a prediction about something the model genuinely computes over, which is probably why any of this worked.
#
- One model family, one scale. 237M parameters, byte-level, English-dominant training. Nothing here has been shown to transfer to a large tokenized model.
- One task family. Invented-word rule chaining is a toy, chosen because it has no world knowledge to leak. It is not “reasoning” in any broad sense.
- Two seeds. Everything above replicates across two seeds, which is our minimum bar, not a strong one. The absolute levels move noticeably between seeds (.53 vs .40 in the early runs); thedirection is what replicates.
- n = 8 on the headline correlation. The convergent agreement across five weighting schemes is the sturdier evidence; the −0.98 itself deserves more words.
- The cross-tokenizer supervision channel is designed, not demonstrated. Coverage arithmetic works today. Routing actual teacher signal through the map is the next build, and it might not pay.
- Multilingual competence is not claimed anywhere. The point of the coverage table is that we can measure what’s missing, not that we’ve filled it.
#
Happy to go deeper on any part of this — especially if you’ve tried cross-tokenizer alignment and hit walls we haven’t hit yet, or if you think the eight-word correlation is doing more work than it should. That last one is a fair hit and we’d rather hear it now.