AI Dojo Open on the typewriterThe typewriter

Explainer

Learning rate

1 min read
Assumes: you know what model weights and a validation set are.
First published 29 October 2026. Last edited 16 November 2026.

How large each weight update is; choose it using validation results.

The learning rate controls how large each weight update is. Too small can make learning slow; too large can make training unstable or damage useful behaviour. Choose it using validation results. Defaults differ by tool and training method: one study found LoRA's best learning rate about ten times higher than full fine-tuning's on its tested tasks (Biderman et al. 2024).

Going deeper

One study found LoRA's best learning rate about ten times higher than full fine-tuning's, recommended sweeping 1e-5 to 5e-4 and picking the highest stable value (Biderman et al. 2024, arxiv.org). Most runs start with a short warmup, raising the rate from near zero, then decay it, linearly by default in Transformers.

In the long read: What happens during training. Every related term is in Fine-tuning, every concept.