Explainer
DPO (direct preference optimization)
Learn from preferred and rejected answer pairs without a separate reward model.
Direct preference optimization (DPO) learns directly from a preferred and a rejected answer to the same prompt. Its original form uses a frozen reference model but skips both a separately trained reward model and the reinforcement-learning loop that generates fresh answers during training (Rafailov et al. 2023). Its loss is classification-like, and a setting called beta controls the strength of the reference constraint. In one controlled comparison, preference-data quality mattered more than the choice of algorithm (Ivison et al. 2024).
Going deeper
DPO can raise the margin between answers by lowering the chosen answer's probability too, especially when the pair differs by only a few tokens (Pal et al. 2024, arxiv.org). Contested: the DPO paper reports matching or beating PPO, while later controlled studies found well-tuned PPO ahead (arxiv.org, arxiv.org). In one comparison, preference data quality mattered more than the algorithm.
In the long read: Beyond examples: preferences and rewards. Every related term is in Fine-tuning, every concept.