Explainer
Synthetic data
Training examples written by a model instead of a person.
Synthetic data is training data a model writes, from questions to whole conversations, which you then filter. It is fast and cheap: Alpaca built 52,000 instruction examples from 175 human-written seeds for under $500 of API cost (crfm.stanford.edu). It also inherits the generating model's blind spots and errors, and filtering matters: one study kept about 9,000 of Alpaca's 52,000 examples and got better results (Chen et al. 2023, arxiv.org). Check the generating model's terms before you train on its output.
Going deeper
Training models over and over on model-made data can make rare patterns fade, called model collapse (Shumailov et al. 2023, arxiv.org); keeping the real data alongside avoided it in one set of experiments (Gerstgrasser et al. 2024, arxiv.org). Keep real data in the mix, spot-check synthetic examples, and record where each one came from.
In the long read: The data decides what it learns. Every related term is in Fine-tuning, every concept.
Read next: Held-out test set