The mechanics that make PPO fine-tuning work — reveals a critic reward coupling inside → InstructGPT's three-step training pipeline

The three-step account describes PPO fine-tuning as maximizing the reward model's score, as though the optimizer that maximizes and the model that scores were independent components. The mechanics theme shows they are not: the PPO value function, the critic supplying the advantage estimate that drives each policy update, is itself initialized from the trained reward model's own weights, not from the SFT model or trained from scratch (instructgpt, §C.4 Details of RLHF training, p. 42). A systematic bias in the reward model's judgments is therefore present twice in the same optimization loop, once as the reward being chased and once as the critic estimating its future value, rather than once as the pipeline narrative's black-box description implies. The mechanics theme also specifies that this optimization is cast as a single-step bandit, one prompt, one response, one terminal reward, a framing the three-step account never states.