AI Dojo Open on the typewriterThe typewriter

Explainer

Epochs and steps

1 min read
Assumes: you know what happens during training.
First published 29 October 2026. Last edited 16 November 2026.

An epoch is one pass over the training data; a step is one weight update.

A step is one update of the weights. An epoch is one complete pass over the whole training set. The microbatch is how many examples one device runs in one forward and backward pass. When the batch you want will not fit in memory, gradient accumulation runs several smaller microbatches and combines their gradients with loss normalization into one weight update. Without that normalization, the update would grow with the number of passes. This saves memory for intermediate results, not for the weights or the optimizer state. Illustration: a microbatch of 2 per GPU, 8 accumulation steps and 4 GPUs make an effective batch of 64 examples per update, so 6,400 examples take 100 steps per epoch. More epochs revisit the same examples, which raises overfitting and forgetting risk. As of 28 October 2026, Transformers (version 5.21.0) defaults to 3 epochs, and Unsloth's guide suggests 1 to 3 (huggingface.co, unsloth.ai).

Going deeper

More epochs mean more passes over the same examples, which raises the risk of overfitting and forgetting. InstructGPT trained for 16 epochs and found that validation loss overfit after 1 while human preference kept improving, so choose by task evaluations (Ouyang et al. 2022, arxiv.org).

In the long read: What happens during training. Every related term is in Fine-tuning, every concept.