PPO-ptx (pretraining gradient mixing) — diverges only off distribution from → PPO (without pretraining mix)

On the paper's own headline metric these two variants are nearly the same model. Adding pretraining updates during PPO "does not lead to large changes in labeler preference" on the API distribution, and on prompts submitted to GPT-3 models PPO-ptx actually performs slightly worse at larger sizes (instructgpt, §"4.1 Results on the API distribution", p. 11). The entire case for preferring PPO-ptx — and it is the variant the paper crowns as "InstructGPT" — rests on evaluations orthogonal to human preference: it recovers most public-NLP-dataset regressions and even surpasses GPT-3 on HellaSwag, where plain PPO regresses (instructgpt, §"4.2 Results on public NLP datasets", p. 15). This makes the pair an unusually clean ablation: γ = 27.8 versus γ = 0 separates what RLHF does to preference from what RLHF does to capabilities, and shows the two effects are nearly independent — the alignment tax can be paid down without touching the alignment gain.