Larger Tokenizers Suffer More Multi-Epoch Regret

A study of byte-level BPE tokenizers found that larger vocabularies suffer more multi-epoch regret: at a 128,000-token vocabulary, training a 300m-parameter model for 8 epochs on a random eighth of FineWeb-Edu text produced a validation loss of 1.4667 versus 1.3854 for a matched 1-epoch run on unique text, a regret of 0.081. The researchers compared vocabularies from 256 bytes up to 128K by truncating the Marin tokenizer (Llama 3's BPE) at K tokens for K from 1K to 64K, measuring validation loss in macro bits per byte across the 16 Paloma domains with untied input and output tables and the MuonH optimizer. The result matters because rare tokens such as " memor" (#16,420) get dedicated embedding and softmax rows in the 128K vocabulary that the 8K vocabulary splits into " mem" and "or", and those rows drive the overfitting gap.

Larger Tokenizers Suffer More Multi-Epoch Regret分词器越大，multi-epoch regret 越高 On a fixed amount of text trained for 8 epochs, larger tokenizers suffer more multi-epoch regret. 同样多的文本训 8 个 epoch，分词器越大，multi-epoch regret 越高。 How Vocabulary Size Shapes Overfitting The full record of this study: code, training curves, and every question we tested, both finished and still running.这项研究的完整记录：代码、运行的训练曲线，还有我们测过的每一个问题（包括已有结论的和还在跑的）。 ../exps/vocab-overfitting.html Byte-level BPE builds a vocabulary by merging frequent pairs字节级 BPE 靠合并高频对建出词表 A tokenizer turns text into token ids from a fixed vocabulary. Each token gets an input embedding row an input vector and an output softmax row weights to predict the next token . Byte-level BPE<sup 1</sup starts from the 256 single bytes. First, a pre-tokenizer splits text into words, numbers, and spaces. Then BPE repeatedly merges the most common adjacent pair until the vocabulary is full. To encode new text, it applies these merges in the same order. The diagram below shows this for the Marin tokenizer Llama 3's BPE, with 128,000 ordinary tokens ranked by merge order . 分词器把文本转成固定词表里的 token 编号。每个 token 在输入端有一行 embedding（输入行），在输出端有一行预测下一个 token 的 softmax 权重（输出行）。字节级 BPE<sup 1</sup 从 256 个单字节起步：预分词器先把文本切成词、数字和空格，BPE 再反复合并出现最多的相邻对，直到填满词表。编码新文本时，就按训练时的顺序依次做这些合并。下面的示意图用 Marin 分词器演示了这个过程（即 Llama 3 的 BPE，有 128,000 个按合并顺序编号的普通 token）。 Keeping only the K lowest-ranked tokens of the 128K vocabulary for K from 1K to 64K is the same as stopping BPE at K tokens. We call any token outside the first 8K a rare token . Take " memor" 16,420 as our running example. The 128K vocabulary gives it its own input and output rows. The 8K vocabulary splits it into " mem" and "or", and the byte vocabulary reads six bytes. Next, we measure what those dedicated rows do across multiple epochs. 只留 128K 词表里编号最小的 K 个 token（K 从 1K 到 64K），就等于让 BPE 在合出 K 个 token 时停手。我们把前 8K 之外的 token 叫作 少见 token 。全文拿 " memor"（ 16,420）举例：128K 给它单独分了输入行和输出行；8K 词表把它拆成 " mem" 和 "or"；字节词表则直接读六个字节。下面我们就来测一测，多训几个 epoch 时，这些专属行会带来什么影响。 Matched 1-epoch and 8-epoch runs compare 256-byte to 128K vocabularies成对跑 1-epoch 和 8-epoch，对比 256 字节到 128K 词表 Training for multiple epochs on a small dataset hurts validation loss on unseen text<sup 2</sup . The opening video shows how we measure this from the 256-byte vocabulary upward. Using FineWeb-Edu documents, we train a matched pair of runs for the same number of steps at each vocabulary size. The 1-epoch run trains for 1 epoch on unique text, while the 8-epoch run trains for 8 epochs on a random eighth of the text. We measure validation loss in macro bits per byte across the 16 Paloma domains, converting loss to bytes so we can compare vocabularies fairly. We define multi-epoch regret as the validation loss of the 8-epoch run, trained on one eighth of the text, minus the validation loss of the 1-epoch run on unique text, with the same steps. The word comes from online learning; here it means a gap in validation loss at the same training step, not a running sum. At 128K, this regret is 1.4667 − 1.3854 = 0.081. We use a 300m model, untied input and output tables, and MuonH, a Muon variant that fixes weight norms on a sphere. Our experiment page ../exps/vocab-overfitting.html has the code and recipe ../exps/vocab-overfitting.html setup and the runs with their training curves ../exps/vocab-overfitting.html record . Next, we track how regret changes as vocabularies add rare tokens like " memor". 拿小份数据训多个 epoch，模型在没见过的文本上验证损失会变差<sup 2</sup 。开头的视频展示了我们怎么从 256 字节词表开始测量这一点。在同样的 FineWeb-Edu 文档上，每个词表都训一对步数相同的模型：1-epoch 跑在不重复文本上训 1 个 epoch；8-epoch 随机抽八分之一文本训 8 个 epoch。我们用 Paloma 16 个领域的宏平均比特/字节来算损失（也就是把损失折算到每个字节上，方便跨词表比较）。我们把 multi-epoch regret 定义为：在八分之一文本上训 8 个 epoch 的验证损失，减去步数相同、在不重复文本上训 1 个 epoch 的验证损失。regret 这个词来自在线学习，但这里指同一训练步上的验证损失之差，不是累积量。在 128K 上，regret 是 1.4667 − 1.3854 = 0.081。实验用 300m 模型、MuonH 优化器（一种把权重范数固定在球面上的 Muon 变体），输入表和输出表不共享。 实验主页 ../exps/vocab-overfitting.html 放了 代码和配方 ../exps/vocab-overfitting.html setup ，还有 运行记录与训练曲线 ../exps/vocab-overfitting.html record 。下面看看词表加入 " memor" 这类少见 token 后，regret 怎么变。 Larger tokenizers suffer more multi-epoch regret at 8 epochs训 8 个 epoch，分词器越大，multi-epoch regret 越高 This study grew out of an earlier experiment. The hashed 2-gram and 3-gram input tables of the Over-Tokenized Transformer https://arxiv.org/abs/2501.16975 Huang et al., 2025 lower the loss up to 520m and overfit when the data repeats. At 1.2B, which trains for 2.4 epochs, the gain becomes a loss. We therefore asked whether the vocabulary of a plain tokenizer has the same problem. 这项研究源于我们更早的一个实验。 Over-Tokenized Transformer https://arxiv.org/abs/2501.16975 （Huang 等人，2025）的哈希 2-gram 和 3-gram 输入表在 520m 及以下能降低损失，但数据一重复就过拟合。到了 1.2B，数据要训 2.4 个 epoch，收益就变成了损失。所以我们想知道，普通分词器的词表是不是也有同样的问题。 Sampled Softmax with Over-Tokenized Transformer The earlier experiment: a sampled softmax from 130m to 1.2B, and the over-tokenized input tables that overfit repeated data.更早的实验：130m 到 1.2B 上的采样 softmax，以及在重复数据上过拟合的 over-tokenized 输入表。 ../exps/sampled-softmax-ov.html Figure 1 shows that multi-epoch regret grows with vocabulary size at 300m. With fixed 4096-token windows dashed line , regret rises from near zero on the 256-byte vocabulary to about 0.08 bits per byte at 128K. Vocabularies from 512 to 8K sit on a flat plateau. Above 8K, every doubling pushes regret higher and wipes out most of 128K's 1-epoch lead over 8K. Over 8 epochs, 128K lowers training loss only slightly more, yet every memorized bit hurts unseen text much more. 图 1 测了 300m 模型下不同词表的 multi-epoch regret。虚线是固定 4096 token 窗口的结果：regret 从 256 字节词表的几乎为零，涨到 128K 的约 0.08 比特/字节。512 到 8K 词表停在一个平缓的平台上。过了 8K，词表每翻一倍，regret 就往上涨一截，几乎吃掉了 128K 在 1 个 epoch 下对 8K 的优势。训 8 个 epoch 时，128K 的训练损失只多降了一点点，但它每多背下 1 比特，在未见文本上损失都多得多。 Even when every window holds the same text ../exps/vocab-overfitting.html h-eqwin solid line , multi-epoch regret still rises above 4K. With 4096-token windows, a smaller vocabulary packs less text per window and takes more steps to read the corpus: 54,247 steps for bytes versus 11,443 for 128K. Matching the text per window puts every run at about 11,444 steps. Then vocabularies from 256 bytes to 4K have a regret of 0.02 to 0.03 bits per byte, so bytes no longer sit near zero. Above 4K, regret still climbs with vocabulary size, from 0.031 at 8K to 0.081 at 128K. With equal text per window, 16K has 0.053 regret and 32K has 0.039, flipping their order from the 4096-token runs. Both runs in a setup use the same subset of documents, and changing the window length keeps only about 9 percent of those documents the same. That document shift may explain why 16K and 32K flip, and why byte regret rises from 0.0003 to 0.0175. 词表越小，4096 token 窗口装的文本就越少，读完同样多的文本要走更多步数：字节词表要走 54,247 步，128K 只要 11,443 步。图里的实线改成让 每个窗口装同样的文本 ../exps/vocab-overfitting.html h-eqwin ，每次运行都走约 11,444 步。这么对齐后，256 字节到 4K 词表的 multi-epoch regret 都在 0.02 到 0.03 比特/字节，字节词表不再接近零。4K 往上，regret 还是随词表变大而上升：8K 是 0.031，128K 是 0.081。装同样文本时，16K 的 regret 是 0.053，32K 是 0.039，顺序跟 4096 token 窗口时反了过来。同一套配置的两次运行重复的是同一批文档，换个窗口长度，重合的文档就只剩约 9%。所以 16K 和 32K 顺序颠倒，还有字节词表从 0.0003 涨到 0.0175，都可能来自换了一批被重复的文档。 In the right panel of Figure 1, we test an over-vocab setup on 128K. It adds hashed 2- and 3-gram input tables 25.7M rows in total that map adjacent token groups to vectors and a second 2-gram output head. Its 1-epoch loss drops by about 0.02 bits per byte, but its multi-epoch regret jumps to 0.71, about nine times 128K's. The same panel shows a one-stage H-Net https://arxiv.org/abs/2507.07955 Hwang, Wang and Gu, 2025 . H-Net learns byte chunks directly from bytes without giving them lookup rows. It aims at the length of 128K tokens 4.74 bytes and reaches about 4 bytes per chunk on the training text. Its regret is 0.031, under two fifths of 128K's 0.081. In a second run that reaches about 6.8 bytes per chunk, longer than 128K's tokens, the regret is 0.015, although the 1-epoch loss is higher 1.432 against 1.373 . So compressing the sequence as far as 128K does not by itself create 128K's regret. 在图 1 右栏，我们给 128K 加上哈希 2-gram 和 3-gram 输入表（共 2570 万行，把相邻两三个 token 哈希到表里查向量），再加第二个预测 2-gram 的输出头，记作 over-vocab。它的 1-epoch 损失降了约 0.02 比特/字节，但 multi-epoch regret 暴涨到 0.71，大概是 128K 的 9 倍。同一栏还画了单阶段 H-Net https://arxiv.org/abs/2507.07955 （Hwang、Wang 和 Gu，2025）。它直接从字节学出块，但不给这些块建查表行。它的目标是 128K token 的长度（4.74 字节），在训练文本上实际约 4 字节一块。它的 regret 只有 0.031，不到 128K（0.081）的五分之二。另一次运行实际约 6.8 字节一块，比 128K 的 token 还长，regret 是 0.015，不过 1-epoch 损失更高（1.432，前者是 1.373）。所以把序列压得和 128K 一样短，本身并不会带来 128K 那样的 regret。 Figure 2 tests 130m and 520m models and finds the same vocabulary ordering as 300m. At 520m, multi-epoch regret wipes out 128K's 1-epoch lead over 8K, leaving them tied. Still, this overall average does not tell us where the regret lands: on common tokens, or on rare tokens like " memor" and unfamiliar domains. 图 2 测了 130m 和 520m 两个规模，词表的排序跟 300m 一样。在 520m 上，multi-epoch regret 抹平了 128K 在 1 个 epoch 下对 8K 的领先，最后两者打平。不过光看整体平均分，还看不出 regret 到底是落在常见 token 上，还是落在 " memor" 这类少见 token 和陌生领域上。 Multi-epoch regret lands on unfamiliar text and rarely seen tokensmulti-epoch regret 集中在陌生文本和很少出现的 token 上 Unfamiliar domains suffer much higher multi-epoch regret at 128K. Following Bergsma et al.<sup 3</sup , Figure 3 breaks regret down by domain. On the 8-epoch training subset, 128K memorizes nearly twice as many bits per byte as the 256-byte vocabulary. On unseen text, 128K's regret grows from about 0.02 bits per byte on unread training-distribution text to about 0.32 on code. Meanwhile, 8K and the 256-byte vocabulary barely move. 照着 Bergsma 等人<sup 3</sup 的做法，图 3 看 multi-epoch regret 落在哪些领域。在训了 8 个 epoch 的那份数据上，128K 每字节背下的比特数快赶上 256 字节词表的两倍。到了没见过的文本上，同分布的未读训练文本只让 128K 多出约 0.02 比特/字节的 regret，到了代码上就涨到约 0.32；而 8K 和 256 字节词表几乎没动。 Much of 128K's multi-epoch regret sits on tokens that the 8-epoch subset rarely holds. Figure 4 groups evaluation positions by how often the target token appears in the 8-epoch subset. Two effects stand out. First, 128K puts many targets into low-count bins, whereas 99.7 percent of byte targets appear over 10,000 times. Second, at the same count from a few hundred upward , 128K has up to twice the regret per byte of 8K. Splitting tokens into ten frequency bins shows the same spike in 128K's rarest tenth, whether grouped by input token ../exps/vocab-overfitting.html fig-freq-in or predicted token ../exps/vocab-overfitting.html fig-freq-out . At 128K, tokens seen fewer than 1,000 times make up 9.5 percent of the text, yet hold 45 percent of the regret. The 8K rank cutoff itself matters less: tokens outside the first 8K are 37 percent of the text and 41 percent of the regret. Half of them appear fewer than 45 times, while " memor" appears 8,849 times. Next, we ask whether the byte vocabulary also degrades if we shrink the subset and run more epochs. 图 4 按目标 token 在 8-epoch 训练子集里的出现次数，把评测位置分组。这里能拆出两件事：第一，128K 把很多目标切进了低频档，而字节词表有 99.7% 的目标出现超过一万次；第二，就算出现次数一样（几百次往上），128K 每字节的 multi-epoch regret 最高也是 8K 的两倍。按训练词频分成十等份，不管是按 输入 token ../exps/vocab-overfitting.html fig-freq-in 还是 预测 token ../exps/vocab-overfitting.html fig-freq-out 分，128K 最少见的十分之一都会冒出同样的尖峰。在 128K 里，出现不到 1,000 次的 token 只占文本的 9.5%，却占了 45% 的 regret。前 8K 这条分界线本身反倒没那么关键：8K 之外的 token 占文本的 37%、占 regret 的 41%，其中一半出现不到 45 次，而 " memor" 其实出现了 8,849 次。看到这儿还有个问题：要是把训练子集缩得更小、训更多 epoch，字节词表自己会不会也跟着变差？ The 256-byte vocabulary holds through 8 epochs and degrades from 16256 字节词表扛得住 8 个 epoch，16 个起开始变差 Does the byte vocabulary ever overfit? Yes, just later. In Figure 5, we fix the step count and shrink the training subset to run more epochs. The 256-byte vocabulary stays flat through 8 epochs and degrades from 16 epochs onward. Still, a smaller vocabulary packs fewer bytes into a window and takes more steps on the corpus. Next, we check whether window length or step count explains the vocabulary gap. 字节词表也会过拟合吗？会，只是来得更晚。图 5 固定总步数，不断缩小训练子集：256 字节词表在 8 个 epoch 以内一直很稳，从 16 个 epoch 开始变差。不过小词表每个窗口装的字节少，跑完语料走的步数多。我们还得排查一下，前面看到的词表差距是不是窗口长度或者步数搞出来的。 Window length and step count do not explain the gap窗口长度和步数解释不了词表差距 Neither a shorter window nor taking more steps explains the multi-epoch regret gap. An 8K token has about a quarter fewer bytes than a 128K token, so an 8K window holds a quarter less text and takes a third more steps. Figure 6 tests both factors. On the left, giving 128K the window bytes or step count of 8K leaves 128K's regret at about 0.08 bits per byte. Also, where in the window the regret sits ../exps/vocab-overfitting.html fig-pos is flat across positions. On the right, stretching the byte window to match a 128K window's text and step count raises 256-byte regret to about 0.02 bits per byte, still under a quarter of 128K's regret. And as Figure 1 showed, at equal steps the byte vocabulary sits level with every vocabulary up to 4K ../exps/vocab-overfitting.html h-q12 . Even with matched windows and steps, though, the main network sees " memor" as one token instead of six bytes. We still need to check whether sequence compression causes the regret. 差距不是窗口更短或步数更多造成的。8K 的每个 token 比 128K 少约四分之一字节，所以 8K 的窗口少装四分之一文本，步数多出三分之一。图 6 排查了这两个因素。左栏把 128K 的窗口字节数或步数改成跟 8K 一样，128K 的 multi-epoch regret 还是停在约 0.08 比特/字节， regret 在窗口各个位置上的分布 ../exps/vocab-overfitting.html fig-pos 也是平的。右栏反过来，把字节窗口拉长，让它装的文本和走的步数都跟 128K 差不多；256 字节词表的 regret 涨到了约 0.02 比特/字节，还是不到 128K 的四分之一。在图 1 里，步数拉平后，字节词表 跟 4K 以内的词表处在同一水平 ../exps/vocab-overfitting.html h-q12 。不过就算对齐了窗口和步数，主干网络看到的 " memor" 还是一个 token，而不是六个字节。还得看看是不是序列压短本身惹的祸。 With smaller tables, 128K's multi-epoch regret nears that of smaller vocabularies把 128K 的表变小，multi-epoch regret 就接近小词表 Does multi-epoch regret come from the vocabulary itself or from the extra embedding parameters? Truncating a vocabulary shrinks both at once. Figure 7 shrinks only the parameters by giving 128K low-rank input and output tables factoring each large table into two smaller matrices . This keeps 128K's tokens, windows, steps, and 8-epoch document subset intact. Tables of rank 96, 192 and 384 ../exps/vocab-overfitting.html h-rank match the parameter counts of the 16K, 32K, and 64K tables. They cut regret to 0.039, 0.053, and 0.074, down from 0.081 for plain 128K. That drop is the firm part. The match to a specific vocabulary size is looser: with equal text per window, 16K and 32K have regrets of 0.053 and 0.039, so the gaps between each rank and its matching vocabulary lie within what a different subset of documents can move. Because the compressed sequences stay identical while regret drops by up to half, sequence compression alone does not set the regret. Also, rank 96 fits unique text 0.043 worse than 16K yet has about the same regret, pointing to parameter count rather than fit quality. Low-rank tables are no fix, though. After 8 epochs, ranks 96 1.507 and 192 1.478 end up worse in validation loss than plain 128K 1.467 , and rank 384 finishes about even 1.471 . Each rank has one run. multi-epoch regret 来自词表变大本身，还是来自多出来的 embedding 参数？直接截断词表会把词表和参数一起变小，分不清是谁的作用。所以图 7 只砍参数：给 128K 换上低秩输入表和输出表（把大矩阵拆成两个小矩阵相乘来省参数），保留原本的 token、窗口、步数和 8-epoch 文档。 秩为 96、192 和 384 的表 ../exps/vocab-overfitting.html h-rank ，参数量分别跟 16K、32K 和 64K 的表一样；它们的 regret 降到了 0.039、0.053 和 0.074，而普通 128K 是 0.081。这个下降是可靠的。但具体跟哪个词表对齐就没那么准了：每个窗口装同样文本时，16K 和 32K 的 regret 是 0.053 和 0.039，所以各个秩跟对应词表的差距，都在换一批文档就能挪动的范围内。序列压得一样短，regret 却最多降了一半，说明光靠序列压短定不了 regret。另外，秩 96 在不重复文本上比 16K 差 0.043，但两者的 regret 差不多，这更支持参数量的解释，而不是拟合好坏的解释。不过低秩表不能拿来救场：训完 8 个 epoch，秩 96（1.507）和 192（1.478）的 8-epoch 验证损失比普通 128K（1.467）更差，秩 384（1.471）与它大致持平。每个秩只有一次运行。 Regularizing rare input rows and weights removes about half of 128K's multi-epoch regret约束少见输入行和权重，能消掉约一半 128K multi-epoch regret We suspect that much of 128K's multi-epoch regret comes from rare tokens' input rows. The intuition is simple: the input row of " memor" updates only when " memor" appears, so across epochs it sees the exact same contexts over and over. To test this, we build each rare input row from the mean of its 8K pieces' rows ../exps/vocab-overfitting.html fig-cvin using " mem" and "or" for " memor" . This removes nearly half of the regret 128K has beyond 8K. Freezing rare input rows after epoch 1 removes nearly two fifths. However, both fixes get much of this by fitting unique text worse: piece-built rows get 1.4092 on unique text versus 1.3854 for plain 128K, while their 8-epoch loss moves only from 1.4667 to 1.4615. Technique D below changes 1-epoch loss by just 0.006. Yet an untargeted control that swaps in random input tokens starting at epoch 5 reaches nearly the same 8-epoch loss 1.446 versus 1.447 . So these runs do not isolate the rare input rows. Figure 8 shows the 81 remedy setups we tested at 128K. 我们猜 128K 的 multi-epoch regret 有很大一部分出在少见 token 的输入行上。道理很简单：" memor" 的输入行只有碰到 " memor" 才更新，多训几个 epoch 就会反复看同一批上下文。为了验证，我们试着 拿 8K 片段输入行的均值拼出少见输入行 ../exps/vocab-overfitting.html fig-cvin （比如 " memor" 就取 " mem" 和 "or" 的均值），这能消掉 128K 超出 8K 那部分 regret 的近一半；在第 1 个 epoch 后冻住少见输入行，也能消掉近五分之二。但这两种固定做法之所以能缩窄差距，很大程度是因为把不重复文本的拟合搞差了：用片段拼出来的输入行在不重复文本上是 1.4092（普通 128K 是 1.3854），而 8-epoch 损失只从 1.4667 微降到 1.4615。下面要讲的技术 D 只让 1-epoch 损失变了 0.006；但我们还跑了个不挑少见 token 的对照，从第 5 个 epoch 起随机换掉输入 token，8-epoch 损失也差不多（1.446 对 1.447）。所以这些实验还没有把少见输入行单独区分出来。图 8 画了我们在 128K 上试过的 81 种补救配置。 A rare token's input row is the mean of its 8K pieces' rows, plus a slowly learned residual.少见 token 的输入行等于它各个 8K 片段输入行的均值，再加一个慢速学习的残差。 Each epoch splits the same text into slightly different tokens, so a rare token does not appear the exact same way 8 times.每个 epoch 把同一段话切成略有不同的 token，免得少见 token 8 次都按同一种切法露面。 Each epoch shuffles the documents and reads them in a new order.每个 epoch 把同一批文档洗牌，换个顺序读。 In training, half of a rare token's occurrences use its 8K pieces' mean instead of its own row; evaluation always uses its own row.训练时，少见 token 有一半机会改用 8K 片段的均值，另一半用自己的输入行；评测时一律用自己的输入行。 Evaluate a moving average of the weights instead of the final weights.评测时不用最后一步的权重，改用权重的滑动平均。 Five techniques drive the progress in Figure 8. Technique A adds a slowly updated residual to each piece-built rare input row. Technique B resamples token splits each epoch using BPE-dropout randomly skipping merges so the same text splits into different tokens without hurting 1-epoch fit. Technique C shuffles document order each epoch. Used alone, Technique A still hurts unique-text fit, and Techniques B and C each cut less than a fifth of the multi-epoch regret. Technique D replaces a rare input row with the mean of its 8K pieces " mem" and "or" for " memor" on half of its training occurrences, blocks gradients to the pieces, and uses the token's own row at evaluation. Most of D's gain comes from blocking the rare row's own gradient on that half of occurrences. The pieces' mean keeps the context intact. Over three runs, random replacement with piece gradients enabled gets 1.4474 after 8 epochs and 1.3916 at 1 epoch; with piece gradients blocked, it gets 1.4440 and 1.3934. 图 8 的进展靠五项技术推进。技术 A 在拼好的少见输入行上加一个慢速更新的残差；技术 B 每个 epoch 用 BPE-dropout（合并时随机跳过一些规则，让同一句话每次切出不同的 token）重新切分，且不伤 1-epoch 拟合；技术 C 每个 epoch 打乱文档顺序。单拉出来用时，技术 A 还是会伤不重复文本的拟合，技术 B 和 C 各自只能消掉不到五分之一的 multi-epoch regret。技术 D 在训练时把少见输入行一半的出现位置换成 8K 片段的均值（" memor" 就换 " mem" 和 "or" 的均值），同时拦住传向片段的梯度，评测时再换回它自己的输入行。主要收益其实来自在那一半位置拦住了少见输入行自己的梯度，片段均值则让上下文保持完整。放开片段梯度时，随机替换跑三次运行的均值是 8-epoch 1.4474、1-epoch 1.3916；拦住梯度后分别是 1.4440 和 1.3934。 Our two best setups cut about half of 128K's multi-epoch regret. The first is Technique E, which adds an exponential moving average of the weights at evaluation on top of C and D. It brings 8-epoch loss down to 1.430, just 0.045 bits per byte above plain 128K's 1-epoch loss on unique text, cutting 45 percent of the regret one run; a close variant with double the learning rate gets 1.432 over two runs, or 43 percent . The second setup combines tied input-output tables, a warmup-stable-decay learning-rate schedule warmup, constant plateau, then decay , and the same weight averaging. It reaches 1.425 in one run, cutting 51 percent of the regret. The gap between 1.425 and 1.430 is within single-run noise, so we treat the two setups as tied. Both slightly hurt unique-text fit 1.3928 and 1.3891 versus 1.3854 , and both leave about half the regret intact. On the output side, freezing rare output rows after epoch 1 ../exps/vocab-overfitting.html fig-frozout hurt 1-epoch fit, while self-distillation and tied tables helped a bit. Our experiment page lists each technique's individual results ../exps/vocab-overfitting.html h-q10 . All these runs use MuonH, which has no weight decay. 目前最好的两套方案能消掉约一半的 128K multi-epoch regret。一套是技术 E：在 C 和 D 的基础上，评测时改用权重的指数滑动平均，把 8-epoch 损失压到 1.430，相对普通 128K 不重复文本的差距只剩 0.045 比特/字节，消掉了 45% 的 regret（一次运行；学习率翻倍的相近变体跑了两次运行，落到 1.432，即消掉 43%）。另一套把共享输入输出表、warmup-stable-decay 学习率调度（先预热、再恒定、最后衰减）和同样的权重平均绑在一起，单次运行落到 1.425，消掉了 51%。1.425 和 1.430 的差距在单次运行的噪声以内，我们算它们打平。这两套方案都会轻微拉低不重复文本的拟合（1.3928 和 1.3891，普通 128K 是 1.3854），也都还剩下一半左右的 regret。在输出端， 第 1 个 epoch 后冻住少见输出行 ../exps/vocab-overfitting.html fig-frozout 会伤 1-epoch 拟合，自蒸馏和共享输入输出表能帮上一点忙。实验主页记下了 每项技术单跑的结果 ../exps/vocab-overfitting.html h-q10 ，全都用的没有权重衰减的 MuonH。 Optimal weight decay grows with the epoch count训练 epoch 越多，最优权重衰减越大 Raising weight decay also cuts multi-epoch regret by about half, and the best weight decay climbs with the epoch count. Because MuonH locks weight norms on a sphere and has no weight decay to tune, Figure 9 switches to standard Muon using the setup of Wen et al.<sup 4</sup Standard Muon also changes the learning-rate schedule, so its losses cannot be compared directly with MuonH. In the left panel, raising weight decay on the 8-epoch run from the 1-epoch best of 0.1 to 0.8 cuts regret by about half, down to 0.055 bits per byte. However, the stronger decay needed for multiple epochs hurts unique-text loss, and even picking the best decay at each epoch count leaves over half the regret. In the right panel, the fitted optimal weight decay grows with the epoch count $R$: from 0.117 at 1 epoch to 0.254 at 4 epochs, scaling roughly as $R^{0.56}$. From 4 to 16 epochs it climbs to 1.87, scaling roughly as $R^{1.44}$, close to the $R^{1.53}$ empirical fit of Baron et al.<sup 5</sup at fixed text. At 32 epochs, loss is still dropping at 6.4, the highest decay we tried, so the optimum sits above 6.4. Zhai et al.<sup 6</sup reported the same direction on ImageNet in 2019. Sweeping the learning rate ../exps/vocab-overfitting.html fig-lr puts the 8-epoch optimum near half the 1-epoch optimum, matching Baron et al. Each rate has one run, though, and single runs cannot tell the 1-epoch rates apart 1.3848 at the base rate versus 1.3841 at double . 直接调大权重衰减也能砍掉约一半 multi-epoch regret，但同样救不回全部。MuonH 把权重范数锁在球面上，没法调权重衰减，所以图 9 换成了 Wen 等人<sup 4</sup 配好的标准 Muon。标准 Muon 顺带改了学习率调度，所以绝对分数不能跟 MuonH 直接比。看左栏，把 8-epoch 的权重衰减从 1-epoch 最优的 0.1 调大到 0.8，regret 能砍掉约一半（降到 0.055 比特/字节）。代价是多 epoch 需要的强衰减会拉差不重复文本的损失，而且就算每个 epoch 数都挑最优衰减，还是会留下一半以上的 regret。看右栏，拟合出来的最优权重衰减随 epoch 数 $R$ 一路涨：1 个 epoch 是 0.117，4 个 epoch 涨到 0.254，大概按 $R^{0.56}$ 增长；从 4 到 16 个 epoch 涨到 1.87，大概按 $R^{1.44}$ 增长，跟 Baron 等人<sup 5</sup 在固定文本量下拟合出的 $R^{1.53}$ 很接近。到了 32 个 epoch，哪怕调到我们试过的最大值 6.4，损失还在往下掉，说明最优值还在 6.4 之上。Zhai 等人<sup 6</sup 早在 2019 年就在 ImageNet 上报告过同样的趋势。另外， 扫学习率 ../exps/vocab-overfitting.html fig-lr 发现 8-epoch 的最优学习率大概是 1-epoch 的一半，这也符合 Baron 等人的结论；不过每个学习率只跑了一次运行，1-epoch 自己的最优点单次运行也分不清（基础学习率 1.3848，翻倍 1.3841）。 Limitations局限性 Our experiments have clear limits. Most runs use a 300m model on FineWeb-Edu for 8 epochs with MuonH, evaluated only on Paloma macro bits per byte. Our smaller BPE vocabularies simply truncate the 128K tokenizer, which also shrinks embedding parameter counts when tables are untied. Both runs in a setup use the same subset of documents, so gaps below about 0.02 bits per byte across vocabularies can come from which documents fall in that subset. Gaps below about 0.01 between single runs sit within the noise. Finally, we pick our best setup using the same Paloma score we report. 说一下实验的局限。大部分运行都只跑了 300m 模型、FineWeb-Edu 数据、8 个 epoch 和 MuonH。打分只看 Paloma 宏平均比特/字节。小号 BPE 词表都是直接截断 128K 分词器得来的，在不共享输入输出表时 embedding 参数也跟着变少。同一套配置的两次运行重复的是同一批文档，所以不同词表之间小于约 0.02 比特/字节的差距，可能只是因为 8-epoch 子集抽到了不同的文档。单次运行之间小于约 0.01 的差距在噪声以内，分不出高下。另外，挑最优配置和报最终结果用的是同一个 Paloma 分数。 Takeaways核心结论 To sum up: a large vocabulary gives rare strings like " memor" their own rows, which helps on unique text, but much of the multi-epoch regret lands on rarely seen tokens. On a 300m model trained for 8 epochs on fixed text with equal text per window, vocabularies from 256 bytes to 4K have 0.02 to 0.03 bits per byte of regret, while 128K BPE has about 0.08. This regret opens up in the second half of training, and much of it sits on tokens that rarely appear in the 8-epoch text. Our two best setups each cut about half of the 128K regret 45 and 51 percent, one run each . 小结一下：大词表给 " memor" 这种少见字符串分了专属行，在不重复文本上确实有用；但多训几个 epoch 后，multi-epoch regret 有很大一部分落在了很少出现的 token 上。300m 模型在固定文本量上训 8 个 epoch、且每个窗口装同样文本时，256 字节到 4K 词表的 regret 是 0.02 到 0.03 比特/字节，128K BPE 约 0.08。这个差距在训练后半程拉开，其中很大一部分落在 8-epoch 文本里少见的 token 上。我们最好的两套方案各消掉了约一半的 128K regret（45% 和 51%，各一次运行）。 Our practical advice: pick your vocabulary size with the epoch count in mind. If you must train a large vocabulary for many epochs, randomly replace rare input rows with the mean of their 8K pieces during training, shuffle the text each epoch, and average the weights. Together, those three steps cut 45 percent of the multi-epoch regret in one run. Alternatively, tying the input and output tables, using a warmup-stable-decay schedule, and averaging the weights cut 51 percent in one run. 给实际训练的建议：定词表大小时，先想想数据要重复训几个 epoch。如果非得拿大词表训很多个 epoch，可以在训练时随机把少见输入行换成 8K 片段的均值、每个 epoch 重新打乱文本，再做权重滑动平均；这三招合起来在一次运行中消掉了 45% 的 multi-epoch regret。或者共享输入输出表、用 warmup-stable-decay 学习率调度再加权重滑动平均，单次运行消掉了 51%。 Open questions开放问题 Beyond these regularization techniques, what else can lower multi-epoch regret in this setup? Even our best setups leave about half of it. 除了这些正则化技术，还有什么办法能在这个设置下降低 multi-epoch regret？最好的方案仍然留下了约一半。 Could a dynamic tokenizer like H-Net work well here? In one run, our one-stage H-Net has under two fifths of 128K's multi-epoch regret, with chunks of about 4 bytes against about 4.7 for 128K's tokens. We leave a detailed study to future work. H-Net 这样的动态分词器会不会很适合这种训练？在一次运行中，我们的单阶段 H-Net 块长约 4 字节（128K 的 token 约 4.7 字节），multi-epoch regret 不到 128K 的五分之二。详细研究留作后续工作。 Acknowledgements致谢 Compute came from Princeton Language and Intelligence on the Della cluster. We thank the Marin project for its codebase, tokenizer, and training recipe, and Xingyu Dang for suggestions on the weight decay runs. 实验算力来自普林斯顿语言与智能中心（PLI）的 Della 集群。感谢 Marin 项目的代码库、分词器和训练配方，感谢 Xingyu Dang 对权重衰减实验的建议。 References参考文献 Citation @misc{chai2026vocabmultiepoch, title = {Larger Tokenizers Suffer More Multi-Epoch Regret}, author = {Chai, Wenhao}, year = {2026}, howpublished = {Blog post}, url = {https://wenhaochai.com/blogs/vocab-multi-epoch.html} }