Explainer
Distillation
Training a smaller student model on a larger teacher model's answers.
Distillation transfers what a big model does into a smaller, cheaper one. The idea goes back to compressing an ensemble of models into one easier to deploy (Hinton et al. 2015, arxiv.org). With language models it usually means sending prompts to a strong teacher, collecting its answers, and fine-tuning a small student on them. A student is not a full copy: models trained to imitate a strong chat model copied its style but closed little to none of the gap on tasks the imitation data did not cover (Gudibande et al. 2023, arxiv.org). Check the teacher's terms: OpenAI's forbid using output to develop models that compete with OpenAI (openai.com).
Going deeper
Contested: the DeepSeek-R1 paper reports that a large model's reasoning patterns can improve smaller models; its distilled models, 1.5 to 70 billion parameters, were fine-tuned for 2 to 3 epochs on 800,000 curated samples (DeepSeek-AI 2025, arxiv.org). Differences in data scale and task coverage likely explain much of the disagreement.
In the long read: The data decides what it learns. Every related term is in Fine-tuning, every concept.
Read next: Synthetic data