AI Dojo Open on the typewriterThe typewriter

Explainer

DPO (direct preference optimization)

1 min read
Assumes: you know what RLHF is.
First published 29 October 2026. Last edited 16 November 2026.

Learn from preferred and rejected answer pairs without a separate reward model.

Direct preference optimization (DPO) learns directly from a preferred and a rejected answer to the same prompt. Its original form uses a frozen reference model but skips both a separately trained reward model and the reinforcement-learning loop that generates fresh answers during training (Rafailov et al. 2023). Its loss is classification-like, and a setting called beta controls the strength of the reference constraint. In one controlled comparison, preference-data quality mattered more than the choice of algorithm (Ivison et al. 2024).

Going deeper

DPO can raise the margin between answers by lowering the chosen answer's probability too, especially when the pair differs by only a few tokens (Pal et al. 2024, arxiv.org). Contested: the DPO paper reports matching or beating PPO, while later controlled studies found well-tuned PPO ahead (arxiv.org, arxiv.org). In one comparison, preference data quality mattered more than the algorithm.

In the long read: Beyond examples: preferences and rewards. Every related term is in Fine-tuning, every concept.