{"slug": "when-a-zero-parameter-cache-overtakes-a-transformer", "title": "When a Zero-Parameter Cache Overtakes a Transformer", "summary": "An engineer's experiment found that a zero-parameter count table over the current document overtakes a 1.43M-parameter transformer with a fixed 64-token window as documents grow longer. At L=1000 tokens the cache reached 0.213 top-1 versus the transformer's 0.149, a 43% relative margin, while adding the transformer on top of corpus plus cache contributed only +0.002. The two mechanisms disagreed on 20-25% of positions, indicating they are genuinely different rather than converging on the same predictions.", "body_md": "The question this series exists to answer, put in a form that can be measured: a transformer sees a fixed 64-token window, a count table over the current document sees the whole document, and as documents get longer, how much of the trained model does the free mechanism replace?\n\nAn earlier experiment had established that the document cache is worth up to +0.106 top-1 on a long coherent document, against +0.014 for every improvement to corpus modelling from Witten-Bell all the way to [a full Gibbs-sampled hierarchical Pitman-Yor model](https://sethwheeler.dev/blog/hpylm-slope/). But that comparison had no neural network in it, so it could not ask the question that matters. This one does: six systems, identical positions, one continuous text cut into documents of length L so that length and coherence grow together. The research repo is not public, so the numbers come from its own harness.\n\nOut of distribution, 1.43M-parameter transformer, 64-token window.\n\n| system | params | L=60 | L=250 | L=1000 | \n|---|---|---|---|---|\n| corpus only, WB-4 on 4M tokens | 0 | 0.128 | 0.129 | 0.124 | \n| **cache only** , this document and nothing else | **0** | 0.103 | **0.172** | **0.213** | \n| corpus + cache, **no neural network** | 0 | 0.139 | 0.201 | 0.228 | \n| **transformer only** | 1.43M | **0.152** | 0.143 | 0.149 | \n| transformer + corpus | 1.43M | 0.147 | 0.156 | 0.153 | \n| transformer + corpus + cache | 1.43M | 0.161 | 0.214 | 0.230 | \n\nPaired tests on the headline comparison, same positions:\n\n```\nL=60    transformer 0.152  cache 0.103  delta -0.050  CI [-0.067,-0.032]  SIGNIFICANT (transformer)\nL=250   transformer 0.143  cache 0.172  delta +0.029  CI [+0.010,+0.049]  SIGNIFICANT (cache)\nL=1000  transformer 0.149  cache 0.213  delta +0.064  CI [+0.044,+0.083]  SIGNIFICANT (cache)\n```\n\nThe test reports both directions, because a confidence interval entirely below zero is a significant win for the transformer, and describing that as \"not significant\" would be wrong. At L=60 the transformer wins and the write-up says so.\n\nThe transformer goes 0.152, 0.143, 0.149 across the three lengths. No trend, and the reason is structural rather than statistical: a 64-token window makes a 1000-token document literally invisible. The cache goes 0.103, 0.172, 0.213; it doubles.\n\nDocument length is a resource one mechanism can spend and the other cannot, and that is the entire result in a line. Everything below is arithmetic on it.\n\nThe crossover is early. Below roughly 100 tokens the trained model wins decisively. Somewhere between 60 and 250 tokens a count table built over the current document, with no parameters, no training and a construction cost in microseconds, overtakes 1.43 million trained parameters, and by 1000 tokens it wins by 0.064 top-1, a 43% relative margin.\n\nThese are also genuinely different mechanisms rather than two variants of one. The two systems disagree on 20 to 25% of positions, against the 3% seen between two hybrids in an earlier experiment. They are not converging on the same predictions by different routes. They disagree, and one of them is right more often as the document grows.\n\nWhat adding the transformer buys on top of corpus plus cache:\n\n|  | L=60 | L=250 | L=1000 | \n|---|---|---|---|\n| gain from adding 1.43M parameters | +0.022 | +0.013 | **+0.002** | \n\nAt L=60 the transformer is carrying real weight; at L=1000 it contributes 0.002 top-1 on top of two count tables, and the substitution is close to complete.\n\nThe mixture weights agree without being told to. The cache's optimal weight climbs from 0.25 to 0.70 to 0.85 as documents lengthen, so at L=1000 the best system is 85% document and 15% corpus. That is a fitted parameter arriving at the same conclusion as the ablation, which is the sort of agreement worth more than either alone.\n\nIt says nothing about a model whose context window covers the document. The comparison is *parameters at a fixed short context* against *a count table over the document*, which is the realistic small-model setting and the one this project's tools actually live in. A long-context model would read the document itself, and this experiment cannot see that at all.\n\nIt also does not escape the finding it sits next to. Counting still does not scale with corpus data; [that axis is closed](https://sethwheeler.dev/blog/hpylm-slope/). What scales here is the amount of *current document* available, which is a different axis, and this experiment says the returns are on the second one.\n\nAnd it is out of distribution by construction: a Gutenberg book against a TinyStories corpus model and a TinyStories transformer. Both trained components face the same shift, so the comparison is internally fair, and an earlier finding already said the cache pays most out of distribution. In-domain the transformer would be stronger and the crossover would move right. Measuring that needs a long-document in-domain corpus, which this project does not have, and it is the clearest gap in this line of work.\n\nTwo smaller caveats belong with the numbers rather than after them. The L=1000 cell has 74 documents and is the noisiest of the three. And the cache at L=60 has a high top-5 of 0.546 alongside its low top-1, because a 60-token document supplies about 30 distinct types, so the distribution is concentrated on a few candidates without being right about which. That is a real property of a short document rather than an artefact, and it inverts as documents lengthen.\n\nThe useful shape here is not \"counting beats neural networks\", which is not what the table says at any length below about 100 tokens. It is that two mechanisms can look like competitors while depending on completely different resources, and that comparing them at one operating point tells you almost nothing.\n\nAt L=60 the honest headline is that a 1.43M-parameter transformer beats a zero-parameter cache by a significant margin. At L=1000 the honest headline is the reverse, by a larger one. Both were measured on the same day with the same harness; either could have been published alone.\n\nThe general version: before comparing two systems, find the axis one of them can spend and the other cannot, then measure along it. If you cannot name that axis, you are probably measuring at whichever point you happened to build first.", "url": "https://wpnews.pro/news/when-a-zero-parameter-cache-overtakes-a-transformer", "canonical_source": "https://dev.to/megapixel99/when-a-zero-parameter-cache-overtakes-a-transformer-37c6", "published_at": "2026-09-13 03:00:00+00:00", "updated_at": "2026-09-13 03:26:23.397624+00:00", "lang": "en", "topics": ["natural-language-processing", "machine-learning", "ai-research", "large-language-models"], "entities": [], "alternates": {"html": "https://wpnews.pro/news/when-a-zero-parameter-cache-overtakes-a-transformer", "markdown": "https://wpnews.pro/news/when-a-zero-parameter-cache-overtakes-a-transformer.md", "text": "https://wpnews.pro/news/when-a-zero-parameter-cache-overtakes-a-transformer.txt", "jsonld": "https://wpnews.pro/news/when-a-zero-parameter-cache-overtakes-a-transformer.jsonld"}}