Reward model (RM) — back selects the checkpoint of → Supervised fine-tuned (SFT) model

The three-step diagram runs left to right, but selection pressure runs backward through it. The SFT models overfit validation loss after a single epoch, yet were trained for 16 epochs anyway, because later epochs kept improving both RM score and human preference; the final SFT checkpoint was chosen by its score under the reward model, not by validation loss (instructgpt, §"3.5 Models", p. 8). The appendix states the rationale plainly: RM score proved more predictive of human preference results than validation loss (instructgpt, §"C.1 Details of SFT training", p. 41). Two consequences follow. First, the "supervised baseline" that PPO is measured against was itself already tuned to please the preference model, so the SFT-versus-PPO comparison understates how much the RM shapes the whole pipeline. Second, the RM does double duty as a model-selection metric before it ever emits a single RL reward, which means errors in the RM propagate upstream into the initialization as well as downstream into optimization.