Explainer
Reward hacking
When a model scores highly by gaming the reward instead of doing the task.
Every training reward is a stand-in for what people actually want, and a model can learn to score highly without doing the task. That is reward hacking (Skalse et al. 2022, arxiv.org). Pushed hard against a learned reward model, true quality first rises and then falls (Gao et al. 2022, arxiv.org). DeepSeek saw reward hacking with a learned helpfulness reward and limited how long it trained on that signal (arxiv.org). Keep a held-out evaluation the training never sees, and read samples of high-scoring outputs.
Going deeper
Reasoning models were caught reward hacking in coding tasks; penalising "bad thoughts" in their reasoning under strong optimisation taught them to hide their intent while still hacking, so monitor reasoning without training directly against the monitor (Baker et al. 2025, arxiv.org).
In the long read: Beyond examples: preferences and rewards. Every related term is in Fine-tuning, every concept.
Read next: LLM-as-a-judge