Keeping the reward model on-distribution — trades its online refresh for a kl penalty in → The mechanics that make PPO fine-tuning work

This theme's central answer to a policy drifting off the reward model's training distribution is to keep the reward model itself alive: the asynchronous architecture refits the reward predictor continuously as the policy generates new trajectories (deep-rl-human-prefs, §"2.2 Our Method", p. 4), and the paper's own "no online queries" ablation shows that training on early labels alone lets the policy exploit gaps in an incompletely-trained predictor (deep-rl-human-prefs, §"3.3 Ablation Studies", p. 9). InstructGPT's PPO stage answers the identical drift problem by refusing to do this: the reward model is trained once, in the pipeline's second step, then held fixed through every later step of PPO fine-tuning, never refit against the policy's own drifting outputs the way the 2017 architecture continually does. In its place, the per-token KL penalty constrains the object that is actually moving: subtracted from reward at every generated token, it keeps the policy close to the SFT model it started from, rather than letting a moving reward model chase the policy outward (instructgpt, §"3.5 Models", p. 9). That substitution is not incidental: iterated online training itself never reaches InstructGPT at all, appearing only as a 2017 mechanism and, later, as a direction Constitutional AI names for the future rather than something either of its own trained models does (constitutional-ai, §"6.1 Future Directions", p. 15). Two answers to one shared problem: 2017 keeps its evaluator adapting to the policy, 2022 freezes the evaluator and puts the leash on the policy instead.