Reward model over-optimization — demands a different mitigation than → Alignment tax

Both pathologies stem from the same act — PPO pushing the policy hard against a learned objective — but the paper treats them as separate diseases with separate cures, and Equation 2 carries one term for each. The per-token KL penalty (coefficient β) exists to mitigate over-optimization of the reward model, while the pretraining-gradient mix (coefficient γ) exists to fix regressions on public NLP datasets (instructgpt, §"3.5 Models", p. 9). The knobs turn out not to be interchangeable: raising β to attack the tax costs validation reward and never fully recovers DROP or SQuADv2, whereas tuning γ reverses the regressions at minimal cost (instructgpt, §"4.2 Results on public NLP datasets", p. 15). The distinction is about what is being lost. Over-optimization loses fidelity to human preference on the training distribution, so its cure is limiting movement; the tax loses capabilities that live off-distribution, so its cure is re-injecting the pretraining signal. Both worsen with more optimization pressure — training for 512k rather than 256k episodes deepens the benchmark regressions (instructgpt, §"E.6 Fixing regressions on public NLP datasets", p. 56).