The question this series exists to answer, put in a form that can be measured: a transformer sees a fixed 64-token window, a count table over the current document sees the whole document, and as documents get longer, how much of the trained model does the free mechanism replace?
An earlier experiment had established that the document cache is worth up to +0.106 top-1 on a long coherent document, against +0.014 for every improvement to corpus modelling from Witten-Bell all the way to a full Gibbs-sampled hierarchical Pitman-Yor model. But that comparison had no neural network in it, so it could not ask the question that matters. This one does: six systems, identical positions, one continuous text cut into documents of length L so that length and coherence grow together. The research repo is not public, so the numbers come from its own harness.
Out of distribution, 1.43M-parameter transformer, 64-token window.
| system | params | L=60 | L=250 | L=1000 |
|---|---|---|---|---|
| corpus only, WB-4 on 4M tokens | 0 | 0.128 | 0.129 | 0.124 |
| cache only , this document and nothing else | 0 | 0.103 | 0.172 | 0.213 |
| corpus + cache, no neural network | 0 | 0.139 | 0.201 | 0.228 |
| transformer only | 1.43M | 0.152 | 0.143 | 0.149 |
| transformer + corpus | 1.43M | 0.147 | 0.156 | 0.153 |
| transformer + corpus + cache | 1.43M | 0.161 | 0.214 | 0.230 |
Paired tests on the headline comparison, same positions:
L=60 transformer 0.152 cache 0.103 delta -0.050 CI [-0.067,-0.032] SIGNIFICANT (transformer)
L=250 transformer 0.143 cache 0.172 delta +0.029 CI [+0.010,+0.049] SIGNIFICANT (cache)
L=1000 transformer 0.149 cache 0.213 delta +0.064 CI [+0.044,+0.083] SIGNIFICANT (cache)
The test reports both directions, because a confidence interval entirely below zero is a significant win for the transformer, and describing that as "not significant" would be wrong. At L=60 the transformer wins and the write-up says so.
The transformer goes 0.152, 0.143, 0.149 across the three lengths. No trend, and the reason is structural rather than statistical: a 64-token window makes a 1000-token document literally invisible. The cache goes 0.103, 0.172, 0.213; it doubles.
Document length is a resource one mechanism can spend and the other cannot, and that is the entire result in a line. Everything below is arithmetic on it.
The crossover is early. Below roughly 100 tokens the trained model wins decisively. Somewhere between 60 and 250 tokens a count table built over the current document, with no parameters, no training and a construction cost in microseconds, overtakes 1.43 million trained parameters, and by 1000 tokens it wins by 0.064 top-1, a 43% relative margin.
These are also genuinely different mechanisms rather than two variants of one. The two systems disagree on 20 to 25% of positions, against the 3% seen between two hybrids in an earlier experiment. They are not converging on the same predictions by different routes. They disagree, and one of them is right more often as the document grows.
What adding the transformer buys on top of corpus plus cache:
| L=60 | L=250 | L=1000 | |
|---|---|---|---|
| gain from adding 1.43M parameters | +0.022 | +0.013 | +0.002 |
At L=60 the transformer is carrying real weight; at L=1000 it contributes 0.002 top-1 on top of two count tables, and the substitution is close to complete.
The mixture weights agree without being told to. The cache's optimal weight climbs from 0.25 to 0.70 to 0.85 as documents lengthen, so at L=1000 the best system is 85% document and 15% corpus. That is a fitted parameter arriving at the same conclusion as the ablation, which is the sort of agreement worth more than either alone.
It says nothing about a model whose context window covers the document. The comparison is parameters at a fixed short context against a count table over the document, which is the realistic small-model setting and the one this project's tools actually live in. A long-context model would read the document itself, and this experiment cannot see that at all.
It also does not escape the finding it sits next to. Counting still does not scale with corpus data; that axis is closed. What scales here is the amount of current document available, which is a different axis, and this experiment says the returns are on the second one.
And it is out of distribution by construction: a Gutenberg book against a TinyStories corpus model and a TinyStories transformer. Both trained components face the same shift, so the comparison is internally fair, and an earlier finding already said the cache pays most out of distribution. In-domain the transformer would be stronger and the crossover would move right. Measuring that needs a long-document in-domain corpus, which this project does not have, and it is the clearest gap in this line of work.
Two smaller caveats belong with the numbers rather than after them. The L=1000 cell has 74 documents and is the noisiest of the three. And the cache at L=60 has a high top-5 of 0.546 alongside its low top-1, because a 60-token document supplies about 30 distinct types, so the distribution is concentrated on a few candidates without being right about which. That is a real property of a short document rather than an artefact, and it inverts as documents lengthen.
The useful shape here is not "counting beats neural networks", which is not what the table says at any length below about 100 tokens. It is that two mechanisms can look like competitors while depending on completely different resources, and that comparing them at one operating point tells you almost nothing.
At L=60 the honest headline is that a 1.43M-parameter transformer beats a zero-parameter cache by a significant margin. At L=1000 the honest headline is the reverse, by a larger one. Both were measured on the same day with the same harness; either could have been published alone.
The general version: before comparing two systems, find the axis one of them can spend and the other cannot, then measure along it. If you cannot name that axis, you are probably measuring at whichever point you happened to build first.