AI Dojo Open on the typewriterThe typewriter

Explainer

Loss masking

1 min read
Assumes: you know what fine-tuning is.
First published 29 October 2026. Last edited 16 November 2026.

Choosing which tokens count toward loss, the training prediction penalty.

A token is a piece of text the model reads or predicts. Training scores how well it predicts the example's next tokens using a penalty called loss. Lower loss means a better fit to those tokens when the data and scoring setup are the same. It is not a percentage, and it does not check whether an answer is true or useful. The training section works through the numbers; evaluation checks the task itself (TRL docs).

Loss masking chooses which tokens count toward that penalty. In assistant-only training, the reply and its end-of-turn marker count, while the prompt does not. The model still reads the prompt as context. Scoring the marker teaches the model when the answer should end. Other setups also score the prompt; assistant-only scoring is a choice for the task, not a universal rule (TRL docs).

Going deeper

Contested: masking is a sensible default, not a law. Also scoring the instruction helped across 21 benchmarks in one study, most when instructions were long, outputs short and examples few (Shi et al. 2024, arxiv.org). If you rely on assistant-only loss, check how many tokens are actually scored before trusting a run.

In the long read: What a training example looks like. Every related term is in Fine-tuning, every concept.