{"slug": "why-a-small-transformer-can-t-copy-a-word-it-hasn-t-seen", "title": "Why a Small Transformer Can't Copy a Word It Hasn't Seen", "summary": "A developer's small transformer that generates web apps from one-line specs fails to generalize to unseen domains, scoring 0% on held-out feature combinations despite fixing data pipeline issues. The model, trained on 4,176 programs, compiles and serves HTTP for seen domains but cannot copy a word it hasn't seen, instead reusing table names from training data. The retraction reveals that the initial claim about amortizing human labor was wrong, and the failure is due to the model's inability to handle out-of-vocabulary tokens.", "body_md": "I have a small transformer that turns a one-line spec into a working web app. It is 11.9M parameters, 6 layers, d=384, trained on 4,176 generated programs. Given *\"a support ticket system with marking a ticket closed, a stats page, searching tickets and creating and viewing tickets\"* it writes a 1,158-token Python file that compiles, serves HTTP, implements search, stats and toggle, and does not implement comments, edit, delete or category. On held-out feature combinations it does that 5 times out of 6.\n\nAsk it for a book catalogue and it writes an appointment booker.\n\nI built this arm to justify a specific claim, and the claim turned out to be wrong. This post is that retraction, plus the three experiments it took to find out *why* the model fails, one of which corrected a diagnosis I had already written down and believed.\n\n(I have written about this project once before, on [how its verification sweep reports on itself](https://sethwheeler.dev/blog/appgen-verification-gap/). That post is about the symbolic half of the same system; this one is about the learned half and does not depend on it. The code is in a private research repo, so there is no link. Every figure below comes from a results file or a command I ran, and I say which.)\n\nThe shipped tool does not use the model at all. It uses compositional synthesis: a parser turns your request into an entity schema plus a feature set, and hand-written emitters assemble exactly that program. It is exact within its grammar and about 2 ms on a CPU.\n\nIts cost is human labour. Each program kind is 160 to 287 lines of hand-written emitter, each language is another 104 to 231, and there is no transfer between them. I wrote in two separate experiment write-ups that this is \"precisely the labour a learned generator would amortise.\" A model, the argument went, would learn the mapping once and cover new cases for free.\n\nFor a long time that was untestable, because the model scored 0%.\n\nFive experiments in a row scored **0/6 compile**. The diagnosis I kept reaching for was capacity or training time, and it was wrong every time. Training to 3.5× lower loss left the failure distribution identical.\n\nWhat actually fixed it was two one-line changes to the data pipeline:\n\n`<unk>`\n\nby construction`<eos>`\n\ntoken.Same architecture, same data, same optimizer. The results file (`021-pipeline-fixes/fixed_combo.json`\n\n) records what happened:\n\n| metric | before | after |\n|---|---|---|\nemitted `<eos>`\n|\nn/a | 6/6 |\n| compile | 0/6 | 6/6 |\n| serves HTTP | 0/6 | 5/6 |\n| all features, strict | 0/6 |\n5/6 (83%) |\n\nStrict means the requested features work *and* the unrequested ones are absent. The model does both. Dev loss improved as well, 0.034 to 0.026, from deleting 4,100 vocabulary entries: entries that can only ever be copied are pure liability, since they consume capacity and are unusable on held-out input.\n\nSo the generator worked, and the claim became testable.\n\nThe `library`\n\ndomain (entity `book`\n\n, table `books`\n\n) was held out of the corpus before any training. If the amortisation argument were right, this is where it would show: a new noun costs the symbolic system a schema synthesiser of about 15 lines, and should cost the model nothing at all.\n\nFrom `025-generator-transfer/fixed_domain.json`\n\n, on 8 held-out requests:\n\n| metric | seen domains | unseen domain |\n|---|---|---|\nemitted `<eos>`\n|\n6/6 | 8/8 |\n| compile | 6/6 | 7/8 |\n| serves HTTP | 5/6 | 3/8 |\nstrict, all features |\n5/6 |\n0/8 |\n| table hit / route hit | 6/6 | 0/8 |\n| mean purity | 1.00 | 0.00 |\n\nZero. Not a degradation, a floor.\n\nI checked what it wrote instead by reading the eight generated files rather than trusting the summary. Every one names a table it was trained on:\n\n```\n000_library_basic.py      contacts\n001_library_delete.py     tickets\n002_library_edit.py       bookings\n003_library_search.py     contacts\n004_library_toggle.py     bookings   (with `tickets` leaking in elsewhere)\n005_library_comments.py   bookings + requests\n006_library_stats.py      bookings\n007_library_category.py   bookings + requests\n```\n\n`bookings`\n\nfor `books`\n\nis the tell: the model reaches for the lexically nearest entity it was trained on. The apps are internally consistent for the wrong entity, which is why 7 of 8 still compile while 0 of 8 are right.\n\nHere is the part that made me stop defending the argument. This is the top of `006_library_stats.py`\n\n, unedited:\n\n```\n# Spec:a simple book lending app offering a stats page and creating and viewing books\n...\n\"\"\"A simple address book app offering a stats page and creating and viewing bookings\"\"\"\nPORT = int(os.environ.get('PORT', '8806'))\nDB = os.environ.get('DB', 'bookings.db')\ncon.execute('CREATE TABLE IF NOT EXISTS bookings (id INTEGER PRIMARY KEY AUTOINCREMENT, guest TEXT, details TEXT)')\n```\n\nLine 1 is the prompt. The model then copies nine consecutive words of it verbatim (*\"offering a stats page and creating and viewing\"*), swaps `book lending`\n\nfor `address book`\n\n, swaps `books`\n\nfor `bookings`\n\n, and gives the table a `guest`\n\ncolumn. The word it needed was in its context, on line 1, and I confirmed that across all eight files: the string `books`\n\nappears exactly once in each, always on line 1, never in the code.\n\nSo the model can copy a nine-word span and cannot copy the one word that decides correctness. The amortisation claim is dead: on the axis where a learned model was supposed to win, it scores 0/8 while 15 lines of symbolic schema synthesis score 10/10.\n\nI wrote down the obvious conclusion: this is a copying problem, and the remedy is pointer attention or byte-level entity tokens.\n\nBefore building any of that, a 30-line probe against the checkpoint (`029-entity-generalization/probe_vocab.py`\n\n) checked whether copying was even the binding constraint. It was not.\n\n** books is not in the 1,817-token vocabulary.** It never occurs in training, so it is\n\n`<unk>`\n\non the way in and unreachable on the way out. Every SQL statement naming it is a single atomic string token, so `'INSERT INTO books (...)'`\n\nis also one out-of-vocabulary token. Only No mechanism defined over that vocabulary could have produced the right program, pointer or otherwise. \"It substitutes a trained entity\" was a symptom of an unreachable output slot, not evidence about copying. That is the fifth representation defect in this line of work, and all five were invisible in the loss curve.\n\nSo: make the noun expressible. Spell the entity at character level (`<w> b o o k s </w>`\n\n) and split literals at the entity boundary so the template fragments stay in vocabulary. Vocabulary went *down*, 1,817 to 1,623, and dev loss was unchanged at 0.027.\n\nIt still scored 0/8 on fidelity. But it failed in a new and much more informative way. At **epoch 1** the model visibly attempted the character channel:\n\n```\nbooooooooooookings\n```\n\nStarting `b`\n\n, `o`\n\n, `o`\n\n, then degenerating into a repetition loop. By **epoch 9** it wrote `bookings`\n\ncleanly, with no character attempt at all. More training made it *abandon* the channel it had started to use.\n\nThat is not a capability failure. It is the objective working as specified. Every training app's entity was one of nine in-vocabulary nouns, so the entity was always predictable from the app's own internal consistency, and spelling a novel noun was never required to be right. Predicting the familiar sequence `b-o-o-k-i-n-g-s`\n\nis simply lower loss. **The model was never asked to copy, so it did not learn to.**\n\nMaking it expressible was necessary and not sufficient. It also cost real quality: in-domain compile fell from 6/6 to 1/6, because spelling an identifier as eight tokens gives eight chances to break it.\n\nThe fix the diagnosis implies is to change the data, not the model: draw a fresh entity noun for every app, so the mapping cannot be memorised and copying out of the spec is the only way to be right.\n\nI regenerated that corpus while writing this. It takes 9 seconds and reproduces byte-for-byte against the tracked manifest:\n\n```\n600 training nouns, 10 held-out nouns, 128 feature combos\nwrote 4240 apps ({'train': 4200, 'test_combo': 20, 'test_entity': 20}), 0 skipped\n```\n\nThe ten held-out nouns include `book`\n\n, so the number stays comparable rather than becoming an easier case, and the script asserts no held-out noun leaks into training.\n\nTrained on that corpus and decoded greedily from the spec alone, **substitution is gone**: 0/10 generations write a trained noun, against 8/8 before. The model now attempts the novel noun. And the new failure names the next blocker precisely:\n\n``` php\nbooks      -> bs\nsprockets  -> sckets\nlectures   -> letes\nharvests   -> heves\n```\n\nIt gets the first character right and the ending right and loses the middle. Spelling a symbol character by character needs state for how far through it you are, and nothing in the architecture carries that. The blocker was never vocabulary, and after the randomised corpus it is no longer distribution; it is positional.\n\nMeanwhile a different line of work in the same repo had a document cache: a count-based model over the text *currently being written*, with no parameters and no training.\n\nCopying a noun out of the prompt is exactly what such a cache is for, and nobody had put the two together. The measurement walks each held-out program left to right and, at every position whose target is part of the entity noun, asks each system to predict it. I re-ran it (`cache_copy.py`\n\n, 63 seconds):\n\n```\nWHOLE-NOUN accuracy where the program FIRST names it:\n  corpus   0/20 apps\n  cache    20/20 apps\n  mix      17/20 apps\n\nsystem       spec (floor)    in the code      first in code\ncorpus             0.243          0.254             0.100\ncache              0.662          0.970             1.000\nmix                0.719          0.978             1.000\n\ncorpus-model errors that were a TRAINED noun's character: 3960\n```\n\n**A cache with zero parameters gets the noun right 20 times out of 20. A count model trained on all 4,200 apps gets it right 0 times out of 20**, and its errors are a trained noun's characters **3,960** times. The corpus arm is the control that makes this mean something: it fails the same way the neural model did, and it is not a data-scale problem, because 4,200 apps is the whole corpus and the score is zero.\n\nThe information was always extractable from the request. What was missing was a mechanism that emits what it just read.\n\nA sweep over the mixing weight is worth putting next to that, because it is the kind of result that flips on the metric you pick. Per-position accuracy peaks at a blend (0.978 at β=0.70) while whole-noun correctness is monotone and peaks at pure cache (20/20 at β=1.00). Blend when the unit of correctness is a token; do not blend when it is a multi-token symbol that is wrong if any character is.\n\nThe honest catch, and it is a big one: all of that is teacher-forced. Under free generation the cache's best weight collapses from about 1.0 to about 0.3, because it begins reading the model's own output rather than true text. Pushed further, it amplifies whatever was just emitted: the gated variant runs degeneracy from 0.34 to **0.92** and takes \"no table at all\" from 1 of 10 to **10 of 10**. **The component that wins on prediction is the one whose input degrades under generation.**\n\nWhat did survive was constraining decoding to spell one of the spans the request actually contains: a trie over the user's own words, again with zero parameters. With the cache at low weight, that takes table hit from 0/10 to **10/10** and purity from 0.00 to **1.00**.\n\nThe obvious objection is that masking to \"spell one of the two nouns in the request\" leaves so little freedom that a model which learned nothing would look correct. That was tested by injecting a second, equally novel, equally legal noun into each request: **9 of 10 chose the requested noun, 1 chose the distractor**, against a chance rate of 0.5 (binomial p = 0.011). The constraint supplies legal spellings; the model supplies which one.\n\nIt fixes the entity, not the program. Whole-app validity is untouched, and a line like `PORT = int(os.environ.get('PORT', deleting them`\n\nin the same outputs is a fair reminder of that.\n\nThree things, and the first one is the one I would actually carry anywhere.\n\n**A 30-line probe beat five experiments of intuition, twice.** \"Needs more capacity\" and \"needs more training\" were wrong at every stage, and the loss curve never once indicated the real defect. Both diagnoses that mattered came from cheap instruments: a teacher-forced per-position probe that localised two pipeline bugs to single token positions, and a vocabulary probe that showed the target string was unreachable before anyone built a copy mechanism for it. Five separate representation defects, all invisible in the loss.\n\n**A model will not learn a capability its training data never requires.** The character channel was available, expressible, and visibly attempted at epoch 1, and training removed it, because being right never depended on it. That is not the model failing to generalise; that is me failing to specify. The fix was a corpus change, and it worked on the first try.\n\n**Check whether the failing thing is even representable before designing the fix.** I had written \"this is a copying problem, use pointer attention\" into my notes as a conclusion. It was a plausible, standard, entirely wrong remedy for a slot that was `<unk>`\n\non the way in and unreachable on the way out. The probe cost 30 lines and saved building the wrong mechanism.\n\nThe arm still does not do what I built it to do. Inside domains it has seen it writes correct software at 83%, which is a genuinely stronger result than I expected from 11.9M parameters, and it is compositional generalisation over 2⁷ feature subsets with no combination-specific code. It just is not the labour-amortising property that would have justified preferring it, and the thing that finally did the copying had no parameters at all.", "url": "https://wpnews.pro/news/why-a-small-transformer-can-t-copy-a-word-it-hasn-t-seen", "canonical_source": "https://dev.to/megapixel99/why-a-small-transformer-cant-copy-a-word-it-hasnt-seen-2fln", "published_at": "2026-08-24 14:00:00+00:00", "updated_at": "2026-08-24 14:13:37.118870+00:00", "lang": "en", "topics": ["machine-learning", "large-language-models", "generative-ai", "ai-research", "developer-tools"], "entities": ["Seth Wheeler"], "alternates": {"html": "https://wpnews.pro/news/why-a-small-transformer-can-t-copy-a-word-it-hasn-t-seen", "markdown": "https://wpnews.pro/news/why-a-small-transformer-can-t-copy-a-word-it-hasn-t-seen.md", "text": "https://wpnews.pro/news/why-a-small-transformer-can-t-copy-a-word-it-hasn-t-seen.txt", "jsonld": "https://wpnews.pro/news/why-a-small-transformer-can-t-copy-a-word-it-hasn-t-seen.jsonld"}}