AI Dojo Open on the typewriterThe typewriter

Explainer

Catastrophic forgetting

1 min read
Assumes: you know what fine-tuning and LoRA are.
First published 29 October 2026. Last edited 16 November 2026.

When training on something new makes a model worse at what it could already do.

Training a model on something new can wear away skills and knowledge it had before. It was observed generally in models from 1 to 7 billion parameters during continued fine-tuning (Luo et al. 2023, arxiv.org), and tends to grow the longer you train; full fine-tuning forgot more than LoRA in one study (Biderman et al. 2024, arxiv.org). You only see it if you test general abilities as well as your task, so keep a fixed set of general checks and run it before and after.

Going deeper

Replay, mixing some earlier or general data into training, is one of the best-evidenced defences: one model kept almost all its earlier performance while learning 8 new tasks with a 1% rehearsal buffer (Scialom et al. 2022, arxiv.org). For large-scale continued pretraining, re-warming the learning rate plus replay matched retraining from scratch (Ibrahim et al. 2024, arxiv.org).

In the long read: What happens during training. Every related term is in Fine-tuning, every concept.