PPO fine-tuning stage (RLHF policy optimization)
Training language models to follow instructions with human feedback — inherited
The third training step, in which the SFT model is further fine-tuned with the Proximal Policy Optimization algorithm to maximize the scalar reward produced by the reward model.
Proximal Policy Optimization is the reinforcement learning algorithm behind the final step of the InstructGPT (2022) pipeline: starting from the supervised model, it nudges the policy toward outputs the reward model scores highly while limiting how far any single update can move.
Sections on the algorithm in general and its role as the third training stage lead into the moving parts that make the stage actually work, each substantial enough to earn its own page further down this wiki.
Proximal Policy Optimization, in general
Proximal Policy Optimization (Schulman et al., 2017) is a policy-gradient reinforcement-learning algorithm that updates a policy while constraining how far a single update can move it from the policy that generated the current batch of data, avoiding the instability of unconstrained large policy updates. Its clipped surrogate objective is $$L^{CLIP}(\theta) = \hat{\mathbb{E}}_t\Big[\min\big(r_t(\theta)\hat{A}_t,\ \text{clip}(r_t(\theta), 1-\epsilon, 1+\epsilon)\,\hat{A}_t\big)\Big], \qquad r_t(\theta) = \frac{\pi_\theta(a_t|s_t)}{\pi_{\theta_{\text{old}}}(a_t|s_t)},$$ where $\hat{A}_t$ is an advantage estimate and $\epsilon$ a small clipping range; the clip term discards the incentive to push $r_t(\theta)$ outside $[1-\epsilon,1+\epsilon]$ once the estimated advantage is already being exploited, keeping each update proximal. PPO predates InstructGPT by five years and is inherited here as the RL algorithm InstructGPT uses to implement its third training step, following the same recipe Stiennon et al. (2020) used for summarization.
As InstructGPT's third stage
PPO fine-tuning takes the Supervised fine-tuned (SFT) model and further trains it by reinforcement learning to maximize the scalar reward produced by the trained Reward model (RM), within a single-step Bandit environment formulation for RLHF: a prompt is presented, the policy generates one response, the reward model scores it, and the episode ends (instructgpt, §"3.5 Models", p. 9). Two anchoring relationships tie this stage to what precedes it. The Supervised fine-tuned (SFT) model — anchors as both init and reference for → PPO fine-tuning stage (RLHF policy optimization) edge notes that the SFT model plays a double role, supplying the policy's initial weights and serving as the reference distribution the Per-token KL penalty from the SFT model measures divergence against, and that these are, subtly, two different SFT checkpoints tuned for different purposes (instructgpt, §"C.3 Details of the initialization models for RLHF", p. 42). The Reward model (RM) — supervises from below → PPO fine-tuning stage (RLHF policy optimization) edge notes that every PPO policy, including the 175B one, is trained against a single 6B reward model, a forced choice since 175B reward-model training proved unstable (instructgpt, §"C.2 Details of RM training", p. 41), which quietly tests whether a smaller judge can usefully steer a larger policy.
Its six working parts
Beyond this high-level story, PPO training is a parent concept whose specific engineering choices, gathered under the The mechanics that make PPO fine-tuning work theme, are each concepts in their own right. The Bandit environment formulation for RLHF supplies the single-step structure described above. The PPO value function, a critic network initialized from the reward model's own weights, and Generalized advantage estimation (GAE), applied here without discounting, together supply the advantage signal the PPO objective needs. The Per-token KL penalty from the SFT model, subtracted per generated token, keeps the policy anchored near the SFT model it started from. Finally, two named model variants instantiate the whole recipe differently: PPO (without pretraining mix) optimizes reward plus KL penalty alone, while PPO-ptx (pretraining gradient mixing) additionally mixes in pretraining-distribution gradients to counteract capability regressions, and it is the latter that the paper calls InstructGPT by default.