# Toy demo: fine-tuning a 3.2M-param char transformer makes it forget Shakespeare (held-out numbers)

> Source: <https://discuss.huggingface.co/t/toy-demo-fine-tuning-a-3-2m-param-char-transformer-makes-it-forget-shakespeare-held-out-numbers/191706#post_1>
> Published: 2026-11-10 18:02:07+00:00

I pre-trained a 3.2M-parameter character-level transformer on Tiny Shakespeare (validation loss 1.50), then fine-tuned it for 1,000 steps on Mrs Beeton’s Book of Household Management (1861).

Results on text held out from both:

After fine-tuning, the prompt “ROMEO:” yields “377. INGREDIENTS.–4 oz. of salt”. A small, concrete example of forgetting.

I also froze the fine-tuned model, hashed its weights (SHA-256), prompted it two ways and hashed again: different outputs, identical hash.

I used both in a 5-minute explainer on training vs fine-tuning vs prompting for a general audience: [https://www.youtube.com/watch?v=hwl4Ve0NBW8](https://www.youtube.com/watch?v=hwl4Ve0NBW8)

It is a toy model and its output is nonsense with the right shape. Happy to answer questions about the setup. What would you use to demonstrate forgetting at this scale?