Distillation takes a large, expensive model and uses it to teach a smaller one. The teacher produces outputs for a body of inputs, and the student trains on those outputs rather than on the original labels.
The reason this works better than training the student from scratch is the shape of the teacher's answer. A hard label says "this is a cat". A teacher's full probability distribution says "cat 0.86, lynx 0.09, dog 0.02", which quietly encodes how the teacher sees the relationships between classes. Hinton's 2015 paper called this the dark knowledge in the teacher's output.
Modern language work has widened the idea. Teams distil reasoning traces, tool-use decisions, and refusal behaviour, not just final tokens. A common pattern is to run an expensive frontier model over a few hundred thousand real production prompts, keep the good responses, and fine-tune a much smaller open model on that set.
The result rarely matches the teacher on hard tasks. It very often matches on the narrow task you actually serve, at a tenth of the cost.