Explainer
RLHF, step by step
Reinforcement learning from human feedback: demonstrations, a learned reward, then reinforcement learning.
In the classic setup for reinforcement learning from human feedback (RLHF), training happens in three stages.
- Train on good demonstrations.
- Learn a reward model from people's rankings of answers.
- Train the assistant toward higher reward scores.
InstructGPT used this recipe, with an algorithm called proximal policy optimization (PPO) for the last stage (Ouyang et al. 2022). The reward model's score stands in for people's judgments; a high score alone does not prove an answer is useful.
Training can push the assistant too far from useful behaviour it already had. A reference model is an unchanged copy kept as a comparison point. A stay-close penalty discourages large changes in answer probabilities relative to that copy. This is called a Kullback-Leibler penalty, or KL penalty. It limits drift, but is not a guarantee of safety or quality (Ouyang et al. 2022).
Going deeper
InstructGPT used about 13,000 supervised prompts, 33,000 reward model prompts and 31,000 PPO prompts. A PPO run typically holds four models: the one being trained, the reference, the reward model and a value model (arxiv.org, arxiv.org). Anthropic ran RLHF as an online loop, refreshing human feedback weekly (Bai et al. 2022, arxiv.org).
In the long read: Beyond examples: preferences and rewards. Every related term is in Fine-tuning, every concept.