{"slug": "just-train-more-measuring-the-exchange-rate", "title": "Just Train More: Measuring the Exchange Rate", "summary": "A developer measured the \"exchange rate\" between training data and document length for a small transformer versus a zero-parameter count-table cache, finding that 16x more training data (from 500K to 8M tokens) moved the crossover point from roughly 74 to 492 tokens of document — a 6.6x shift, or about 1.60x per doubling of training data. Extrapolating that rate, matching the cache on a 4000-token file would require roughly 10 to 20 times more in-domain code than CPython's standard library and PyTorch combined, which the developer argues makes the \"just train more\" objection directionally right but quantitatively weak for per-repo completion.", "body_md": "[The previous experiment](https://sethwheeler.dev/blog/document-vs-parameters/) found that a count table over the current document overtakes a small transformer at a few hundred tokens of document, and that the transformer's marginal contribution collapses past that point. There is one obvious objection and it deserves a measurement rather than an argument:\n\nJust train more. 8M tokens is nothing. Of course a 2.45M-parameter transformer loses; train it on a real corpus and the crossover disappears.\n\nThe objection is directionally right and quantitatively weak, and the useful part is the rate. The research repo is not public, so the figures come from its own harness.\n\nAn earlier experiment had saved checkpoints at 500K, 2M and 8M tokens of code with identical architecture, so the sweep is a re-evaluation rather than a training run. The cache uses no training data at all, which makes its accuracy a constant across the sweep: whatever the transformer gains from 16x more data is exactly what moves the crossover.\n\nThere is one wrinkle worth stating because it could have quietly broken the comparison. Each checkpoint built its own 16K vocabulary from its own subset, and only 39.8% of the 500K and 8M vocabularies overlap. So every cell is encoded with the vocabulary of the checkpoint it tests, and the corpus count model is built on the same tokens the transformer was trained on. Comparisons are always *within* one data scale; what gets compared across scales is only the crossover position.\n\nIn-distribution held-out code. The value is the paired delta, cache minus transformer, so positive means the zero-parameter cache wins.\n\n```\n  train             250      400      600      800     1000     4000\n  500,000        0.073*   0.123*   0.124*   0.193*   0.196*   0.268*\n  2,000,000      0.022    0.077*   0.078*   0.151*   0.135*   0.226*\n  8,000,000     -0.057*   0.006   -0.009    0.055*   0.044*   0.098*\n  (* = paired-significant)\n```\n\nThe cache's own top-1 is 0.311, 0.312, 0.313 at L=250 and 0.487, 0.488, 0.487 at L=4000 across the three rows. It is supposed to be constant, because it never sees the training corpus, and it is, to within a vocabulary difference. That invariance is the experiment's internal control: every bit of movement in the table is the trained component improving, and if the cache row had drifted, nothing else here would be readable.\n\nA log-linear fit of each row's delta against document length:\n\n| training tokens | crossover | \n|---|---|\n| 500K | ~74 tokens of document | \n| 2M | ~155 | \n| 8M | ~492 | \n\nSixteen times the training data moved the crossover 6.6x, which is document length times 1.60 per doubling of training data. Extrapolating that rate, which is illustrative rather than measured, since this project's own earlier work is a warning about trusting a slope past its fitted range:\n\n| to be competitive at | needs | \n|---|---|\n| 1000 tokens of open file | ~23M tokens (3x) | \n| 4000 tokens | ~174M tokens (22x) | \n| 16000 tokens | ~1.3B tokens (167x) | \n\nUsing only the 2M to 8M segment, where the model is less undertrained, the rate is 1.78 per doubling and L=4000 needs about 97M tokens, so 12x. The honest range for matching a zero-parameter cache on a 4000-token file is therefore roughly 10 to 20 times more in-domain code than CPython's standard library and PyTorch combined.\n\nThat is what makes the objection weak rather than wrong. The crossover does move. It moves at a price, and per-repo completion is precisely the setting where the code to pay it does not exist.\n\nTraining data lifts the transformer's flat line. It does not give the model access to the document. A 64-token window cannot see a 4000-token file no matter how much text it was trained on, so the transformer's gain from data is roughly the same at every document length: +0.132 at L=250, +0.148 at L=1000, +0.170 at L=4000 across the full 16x.\n\nThe crossover moves only because the flat line rises, which makes the exchange rate a race between two slopes that have nothing to do with each other. They turn out to be the same size:\n\nSixteen times the training corpus buys about as much as letting a zero-parameter mechanism read the rest of the file it is already sitting in. One of those is a data-collection project; the other is already in the editor's buffer.\n\nAt 8M tokens and L=250 the corpus count table scores 0.415 against the transformer's 0.370. That reproduces an earlier finding, that counting beats the small transformer on code, on a different harness, which is the sort of accidental replication worth keeping.\n\nCombined with the corpus-plus-cache numbers from a neighbouring experiment, 0.461 rising to 0.551 across these lengths, the transformer does not beat the free components at *any* document length in this range. It only ever earns its place as a mixture component, at a weight of about 0.2.\n\nThe objection had two halves and this addresses one. The other one is that **parameters and context window are confounded in every experiment in this line.** Everything here varies training data at a fixed 64-token window, and the cache's advantage is entirely about seeing text the transformer cannot see. A model with a 4096-token window would read the document itself.\n\nThat is the experiment this line of work still owes, and until it is run, the correct summary is narrow: at a fixed short context, buying document access is far cheaper than buying training data, and the exchange rate is measured rather than argued.\n\nThree limits on the numbers themselves. The middle checkpoint is the weak link, because the transformer was nearly flat from 500K to 2M and jumped at 8M, so the global fit averages a slow segment with a fast one; both segment rates are given above for that reason. Everything is in-distribution, and cross-project the cache wins earlier, so this is the conservative split for the cache. And the chunking takes text from the start of each file, so larger L also means positions deeper in a file: the within-cell comparisons that make up the whole table are clean, but a single system's trend across L is not.", "url": "https://wpnews.pro/news/just-train-more-measuring-the-exchange-rate", "canonical_source": "https://dev.to/megapixel99/just-train-more-measuring-the-exchange-rate-1bp0", "published_at": "2026-09-14 03:00:00+00:00", "updated_at": "2026-09-14 03:26:36.209749+00:00", "lang": "en", "topics": ["machine-learning", "large-language-models", "ai-research", "developer-tools"], "entities": ["CPython", "PyTorch"], "alternates": {"html": "https://wpnews.pro/news/just-train-more-measuring-the-exchange-rate", "markdown": "https://wpnews.pro/news/just-train-more-measuring-the-exchange-rate.md", "text": "https://wpnews.pro/news/just-train-more-measuring-the-exchange-rate.txt", "jsonld": "https://wpnews.pro/news/just-train-more-measuring-the-exchange-rate.jsonld"}}