PPO (without pretraining mix)
Training language models to follow instructions with human feedback — introduced
The RLHF policy variant trained purely to maximize reward-model score plus a KL penalty, with the pretraining-gradient coefficient set to zero, used as an ablation against PPO-ptx.
When InstructGPT's authors wanted to know what their pretraining-gradient trick actually bought them, they needed a version trained without it. This is that variant: plain reinforcement learning against the reward model, penalized only for drifting from its starting point.
The page first pins down the ablation exactly, then explains what it isolates. From there, the natural next stop is PPO-ptx, the sibling variant it exists to be measured against.
The ablation, precisely
PPO (without pretraining mix) is InstructGPT's pure RLHF policy: it maximizes reward-model score plus the per-token Per-token KL penalty from the SFT model, with the pretraining-loss coefficient $\gamma$ in the PPO objective set to exactly zero (instructgpt, §"3.5 Models", p. 9; Equation 2). It is a child concept of PPO fine-tuning stage (RLHF policy optimization), introduced specifically as the ablation baseline against which PPO-ptx (pretraining gradient mixing), the paper's default "InstructGPT," is measured (instructgpt, §"4.1 Results on the API distribution", p. 11).
What the ablation isolates
The PPO-ptx (pretraining gradient mixing) — diverges only off distribution from → PPO (without pretraining mix) edge reports that on the paper's own headline metric, human preference on the API distribution, PPO and PPO-ptx are nearly the same model; adding pretraining gradients "does not lead to large changes in labeler preference" (instructgpt, §"4.1 Results on the API distribution", p. 11). The entire case for preferring PPO-ptx rests on evaluations orthogonal to preference: PPO shows larger regressions on public NLP benchmarks than PPO-ptx, illustrating the Alignment tax before mitigation, while PPO-ptx recovers most of that ground and even surpasses GPT-3 on HellaSwag (instructgpt, §"4.2 Results on public NLP datasets", p. 15). This makes the PPO/PPO-ptx pair an unusually clean natural experiment within the The mechanics that make PPO fine-tuning work theme: it separates what RLHF does to human preference from what RLHF does to general capability, and shows the two effects can be nearly decoupled, the alignment tax paid down without touching the alignment gain.