Why a Small Transformer Can't Copy a Word It Hasn't Seen

A developer's small transformer that generates web apps from one-line specs fails to generalize to unseen domains, scoring 0% on held-out feature combinations despite fixing data pipeline issues. The model, trained on 4,176 programs, compiles and serves HTTP for seen domains but cannot copy a word it hasn't seen, instead reusing table names from training data. The retraction reveals that the initial claim about amortizing human labor was wrong, and the failure is due to the model's inability to handle out-of-vocabulary tokens.

I have a small transformer that turns a one-line spec into a working web app. It is 11.9M parameters, 6 layers, d=384, trained on 4,176 generated programs. Given "a support ticket system with marking a ticket closed, a stats page, searching tickets and creating and viewing tickets" it writes a 1,158-token Python file that compiles, serves HTTP, implements search, stats and toggle, and does not implement comments, edit, delete or category. On held-out feature combinations it does that 5 times out of 6. Ask it for a book catalogue and it writes an appointment booker. I built this arm to justify a specific claim, and the claim turned out to be wrong. This post is that retraction, plus the three experiments it took to find out why the model fails, one of which corrected a diagnosis I had already written down and believed. I have written about this project once before, on how its verification sweep reports on itself https://sethwheeler.dev/blog/appgen-verification-gap/ . That post is about the symbolic half of the same system; this one is about the learned half and does not depend on it. The code is in a private research repo, so there is no link. Every figure below comes from a results file or a command I ran, and I say which. The shipped tool does not use the model at all. It uses compositional synthesis: a parser turns your request into an entity schema plus a feature set, and hand-written emitters assemble exactly that program. It is exact within its grammar and about 2 ms on a CPU. Its cost is human labour. Each program kind is 160 to 287 lines of hand-written emitter, each language is another 104 to 231, and there is no transfer between them. I wrote in two separate experiment write-ups that this is "precisely the labour a learned generator would amortise." A model, the argument went, would learn the mapping once and cover new cases for free. For a long time that was untestable, because the model scored 0%. Five experiments in a row scored 0/6 compile . The diagnosis I kept reaching for was capacity or training time, and it was wrong every time. Training to 3.5× lower loss left the failure distribution identical. What actually fixed it was two one-line changes to the data pipeline: <unk by construction <eos token.Same architecture, same data, same optimizer. The results file 021-pipeline-fixes/fixed combo.json records what happened: | metric | before | after | |---|---|---| emitted <eos | n/a | 6/6 | | compile | 0/6 | 6/6 | | serves HTTP | 0/6 | 5/6 | | all features, strict | 0/6 | 5/6 83% | Strict means the requested features work and the unrequested ones are absent. The model does both. Dev loss improved as well, 0.034 to 0.026, from deleting 4,100 vocabulary entries: entries that can only ever be copied are pure liability, since they consume capacity and are unusable on held-out input. So the generator worked, and the claim became testable. The library domain entity book , table books was held out of the corpus before any training. If the amortisation argument were right, this is where it would show: a new noun costs the symbolic system a schema synthesiser of about 15 lines, and should cost the model nothing at all. From 025-generator-transfer/fixed domain.json , on 8 held-out requests: | metric | seen domains | unseen domain | |---|---|---| emitted <eos | 6/6 | 8/8 | | compile | 6/6 | 7/8 | | serves HTTP | 5/6 | 3/8 | strict, all features | 5/6 | 0/8 | | table hit / route hit | 6/6 | 0/8 | | mean purity | 1.00 | 0.00 | Zero. Not a degradation, a floor. I checked what it wrote instead by reading the eight generated files rather than trusting the summary. Every one names a table it was trained on: 000 library basic.py contacts 001 library delete.py tickets 002 library edit.py bookings 003 library search.py contacts 004 library toggle.py bookings with tickets leaking in elsewhere 005 library comments.py bookings + requests 006 library stats.py bookings 007 library category.py bookings + requests bookings for books is the tell: the model reaches for the lexically nearest entity it was trained on. The apps are internally consistent for the wrong entity, which is why 7 of 8 still compile while 0 of 8 are right. Here is the part that made me stop defending the argument. This is the top of 006 library stats.py , unedited: Spec:a simple book lending app offering a stats page and creating and viewing books ... """A simple address book app offering a stats page and creating and viewing bookings""" PORT = int os.environ.get 'PORT', '8806' DB = os.environ.get 'DB', 'bookings.db' con.execute 'CREATE TABLE IF NOT EXISTS bookings id INTEGER PRIMARY KEY AUTOINCREMENT, guest TEXT, details TEXT ' Line 1 is the prompt. The model then copies nine consecutive words of it verbatim "offering a stats page and creating and viewing" , swaps book lending for address book , swaps books for bookings , and gives the table a guest column. The word it needed was in its context, on line 1, and I confirmed that across all eight files: the string books appears exactly once in each, always on line 1, never in the code. So the model can copy a nine-word span and cannot copy the one word that decides correctness. The amortisation claim is dead: on the axis where a learned model was supposed to win, it scores 0/8 while 15 lines of symbolic schema synthesis score 10/10. I wrote down the obvious conclusion: this is a copying problem, and the remedy is pointer attention or byte-level entity tokens. Before building any of that, a 30-line probe against the checkpoint 029-entity-generalization/probe vocab.py checked whether copying was even the binding constraint. It was not. books is not in the 1,817-token vocabulary. It never occurs in training, so it is <unk on the way in and unreachable on the way out. Every SQL statement naming it is a single atomic string token, so 'INSERT INTO books ... ' is also one out-of-vocabulary token. Only No mechanism defined over that vocabulary could have produced the right program, pointer or otherwise. "It substitutes a trained entity" was a symptom of an unreachable output slot, not evidence about copying. That is the fifth representation defect in this line of work, and all five were invisible in the loss curve. So: make the noun expressible. Spell the entity at character level <w b o o k s </w and split literals at the entity boundary so the template fragments stay in vocabulary. Vocabulary went down , 1,817 to 1,623, and dev loss was unchanged at 0.027. It still scored 0/8 on fidelity. But it failed in a new and much more informative way. At epoch 1 the model visibly attempted the character channel: booooooooooookings Starting b , o , o , then degenerating into a repetition loop. By epoch 9 it wrote bookings cleanly, with no character attempt at all. More training made it abandon the channel it had started to use. That is not a capability failure. It is the objective working as specified. Every training app's entity was one of nine in-vocabulary nouns, so the entity was always predictable from the app's own internal consistency, and spelling a novel noun was never required to be right. Predicting the familiar sequence b-o-o-k-i-n-g-s is simply lower loss. The model was never asked to copy, so it did not learn to. Making it expressible was necessary and not sufficient. It also cost real quality: in-domain compile fell from 6/6 to 1/6, because spelling an identifier as eight tokens gives eight chances to break it. The fix the diagnosis implies is to change the data, not the model: draw a fresh entity noun for every app, so the mapping cannot be memorised and copying out of the spec is the only way to be right. I regenerated that corpus while writing this. It takes 9 seconds and reproduces byte-for-byte against the tracked manifest: 600 training nouns, 10 held-out nouns, 128 feature combos wrote 4240 apps {'train': 4200, 'test combo': 20, 'test entity': 20} , 0 skipped The ten held-out nouns include book , so the number stays comparable rather than becoming an easier case, and the script asserts no held-out noun leaks into training. Trained on that corpus and decoded greedily from the spec alone, substitution is gone : 0/10 generations write a trained noun, against 8/8 before. The model now attempts the novel noun. And the new failure names the next blocker precisely: php books - bs sprockets - sckets lectures - letes harvests - heves It gets the first character right and the ending right and loses the middle. Spelling a symbol character by character needs state for how far through it you are, and nothing in the architecture carries that. The blocker was never vocabulary, and after the randomised corpus it is no longer distribution; it is positional. Meanwhile a different line of work in the same repo had a document cache: a count-based model over the text currently being written , with no parameters and no training. Copying a noun out of the prompt is exactly what such a cache is for, and nobody had put the two together. The measurement walks each held-out program left to right and, at every position whose target is part of the entity noun, asks each system to predict it. I re-ran it cache copy.py , 63 seconds : WHOLE-NOUN accuracy where the program FIRST names it: corpus 0/20 apps cache 20/20 apps mix 17/20 apps system spec floor in the code first in code corpus 0.243 0.254 0.100 cache 0.662 0.970 1.000 mix 0.719 0.978 1.000 corpus-model errors that were a TRAINED noun's character: 3960 A cache with zero parameters gets the noun right 20 times out of 20. A count model trained on all 4,200 apps gets it right 0 times out of 20 , and its errors are a trained noun's characters 3,960 times. The corpus arm is the control that makes this mean something: it fails the same way the neural model did, and it is not a data-scale problem, because 4,200 apps is the whole corpus and the score is zero. The information was always extractable from the request. What was missing was a mechanism that emits what it just read. A sweep over the mixing weight is worth putting next to that, because it is the kind of result that flips on the metric you pick. Per-position accuracy peaks at a blend 0.978 at β=0.70 while whole-noun correctness is monotone and peaks at pure cache 20/20 at β=1.00 . Blend when the unit of correctness is a token; do not blend when it is a multi-token symbol that is wrong if any character is. The honest catch, and it is a big one: all of that is teacher-forced. Under free generation the cache's best weight collapses from about 1.0 to about 0.3, because it begins reading the model's own output rather than true text. Pushed further, it amplifies whatever was just emitted: the gated variant runs degeneracy from 0.34 to 0.92 and takes "no table at all" from 1 of 10 to 10 of 10 . The component that wins on prediction is the one whose input degrades under generation. What did survive was constraining decoding to spell one of the spans the request actually contains: a trie over the user's own words, again with zero parameters. With the cache at low weight, that takes table hit from 0/10 to 10/10 and purity from 0.00 to 1.00 . The obvious objection is that masking to "spell one of the two nouns in the request" leaves so little freedom that a model which learned nothing would look correct. That was tested by injecting a second, equally novel, equally legal noun into each request: 9 of 10 chose the requested noun, 1 chose the distractor , against a chance rate of 0.5 binomial p = 0.011 . The constraint supplies legal spellings; the model supplies which one. It fixes the entity, not the program. Whole-app validity is untouched, and a line like PORT = int os.environ.get 'PORT', deleting them in the same outputs is a fair reminder of that. Three things, and the first one is the one I would actually carry anywhere. A 30-line probe beat five experiments of intuition, twice. "Needs more capacity" and "needs more training" were wrong at every stage, and the loss curve never once indicated the real defect. Both diagnoses that mattered came from cheap instruments: a teacher-forced per-position probe that localised two pipeline bugs to single token positions, and a vocabulary probe that showed the target string was unreachable before anyone built a copy mechanism for it. Five separate representation defects, all invisible in the loss. A model will not learn a capability its training data never requires. The character channel was available, expressible, and visibly attempted at epoch 1, and training removed it, because being right never depended on it. That is not the model failing to generalise; that is me failing to specify. The fix was a corpus change, and it worked on the first try. Check whether the failing thing is even representable before designing the fix. I had written "this is a copying problem, use pointer attention" into my notes as a conclusion. It was a plausible, standard, entirely wrong remedy for a slot that was <unk on the way in and unreachable on the way out. The probe cost 30 lines and saved building the wrong mechanism. The arm still does not do what I built it to do. Inside domains it has seen it writes correct software at 83%, which is a genuinely stronger result than I expected from 11.9M parameters, and it is compositional generalisation over 2⁷ feature subsets with no combination-specific code. It just is not the labour-amortising property that would have justified preferring it, and the thing that finally did the copying had no parameters at all.