Toy demo: fine-tuning a 3.2M-param char transformer makes it forget Shakespeare (held-out numbers)

I pre-trained a 3.2M-parameter character-level transformer on Tiny Shakespeare (validation loss 1.50), then fine-tuned it for 1,000 steps on Mrs Beeton’s Book of Household Management (1861).

Results on text held out from both:

After fine-tuning, the prompt “ROMEO:” yields “377. INGREDIENTS.–4 oz. of salt”. A small, concrete example of forgetting.

I also froze the fine-tuned model, hashed its weights (SHA-256), prompted it two ways and hashed again: different outputs, identical hash.

I used both in a 5-minute explainer on training vs fine-tuning vs prompting for a general audience: https://www.youtube.com/watch?v=hwl4Ve0NBW8 It is a toy model and its output is nonsense with the right shape. Happy to answer questions about the setup. What would you use to demonstrate forgetting at this scale?