Toy demo: fine-tuning a 3.2M-param char transformer makes it forget Shakespeare (held-out numbers)
A 3.2M-parameter character-level transformer pre-trained on Tiny Shakespeare to a validation loss of 1.50 forgot its original training after 1,000 fine-tuning steps on Mrs Beeton's Book of Household Management (1861), wi…