Explainer
Continued pretraining
More next-word training on raw text from one field, to adapt a model to its language.
Continued pretraining keeps doing what pretraining did, predicting the next word, but on a large amount of text from one field such as medicine or law. It needs no question and answer pairs, just in-domain text, and improved results across biomedical, computer science, news and review tasks in one study (Gururangan et al. 2020, arxiv.org). In a comparison of knowledge-intensive tasks, retrieval outperformed further training on raw documents in the tested setups; that is evidence for trying retrieval first, not a result for every task (Ovadia et al. 2023, arxiv.org).
Going deeper
It usually needs far more text and compute than supervised fine-tuning. At large scale, re-warming and re-decaying the learning rate and replaying some earlier data matched retraining from scratch (Ibrahim et al. 2024, arxiv.org). Hosted support varies: as of 28 October 2026, Amazon Bedrock lists supervised fine-tuning, reinforcement fine-tuning and distillation, not continued pretraining (docs.aws.amazon.com).
In the long read: Should you fine-tune at all?. Every related term is in Fine-tuning, every concept.