Distillation

Also known as: knowledge distillation, model distillation

Knowledge distillation trains a smaller student model to mimic the output distribution of a larger teacher model, transferring capability while dramatically reducing inference cost. Instead of training the student only on hard labels, distillation uses the teacher's soft probability outputs — or intermediate representations — as a rich training signal. Distillation is widely used to produce efficient deployment models and can be combined with fine-tuning to adapt a student to a specific domain while retaining quality from a frontier teacher.