Two things that surprised me when I distilled vision models:
The temperature is doing double duty. High temperature softens the teacher’s logits so the student learns the dark knowledge — which wrong classes look plausible, not just the right answer. That inter-class similarity is the actual transfer; hard labels never carry it. Too hot and it all becomes uniform noise the student memorizes. Somewhere around T=3-4 is usually where the dark knowledge is legible without washing out.
The student inherits the teacher’s calibration, not just its accuracy. If the teacher is overconfident (most are), the student comes out overconfident too — and no serving-time softmax temperature fixes what the training temperature already baked in. Worth checking ECE on the student, not just accuracy, or the “smaller model” quietly becomes a confidently-wrong one.
The non-obvious one from my side: distilling across architectures hurts more than the papers suggest. ResNet-18 from a ResNet-50 teacher works because the inductive biases rhyme. Cross-family (CNN teacher, ViT student) needs feature-level hints, not just logits — the logit surface alone doesn’t carry enough of the “how to see” part.
Your pruning-then-distilling combo is interesting — recovering into the sparse structure usually beats distilling first and pruning after, because the student gets to learn the teacher’s behavior on the exact structure it will serve with.