Explainer
Catastrophic forgetting
When training on something new makes a model worse at what it could already do.
Training a model on something new can wear away skills and knowledge it had before. It was observed generally in models from 1 to 7 billion parameters during continued fine-tuning (Luo et al. 2023, arxiv.org), and tends to grow the longer you train; full fine-tuning forgot more than LoRA in one study (Biderman et al. 2024, arxiv.org). You only see it if you test general abilities as well as your task, so keep a fixed set of general checks and run it before and after.
Going deeper
Replay, mixing some earlier or general data into training, is one of the best-evidenced defences: one model kept almost all its earlier performance while learning 8 new tasks with a 1% rehearsal buffer (Scialom et al. 2022, arxiv.org). For large-scale continued pretraining, re-warming the learning rate plus replay matched retraining from scratch (Ibrahim et al. 2024, arxiv.org).
In the long read: What happens during training. Every related term is in Fine-tuning, every concept.
Read next: Held-out test set