Explainer
Learning rate
How large each weight update is; choose it using validation results.
The learning rate controls how large each weight update is. Too small can make learning slow; too large can make training unstable or damage useful behaviour. Choose it using validation results. Defaults differ by tool and training method: one study found LoRA's best learning rate about ten times higher than full fine-tuning's on its tested tasks (Biderman et al. 2024).
Going deeper
One study found LoRA's best learning rate about ten times higher than full fine-tuning's, recommended sweeping 1e-5 to 5e-4 and picking the highest stable value (Biderman et al. 2024, arxiv.org). Most runs start with a short warmup, raising the rate from near zero, then decay it, linearly by default in Transformers.
In the long read: What happens during training. Every related term is in Fine-tuning, every concept.
Read next: Epochs and steps