Self-Specialized Teachers for Domain Post-Training
Researchers propose self-specialized teacher distillation (SSTD), a two-stage procedure that trains a domain teacher from a base model and distills its token distribution to a student on self-sampled prefixes, improving …