PPO-ptx (pretraining gradient mixing) — outperforms tightening of → Per-token KL penalty from the SFT model

Both of these are knobs on the same objective (β and γ in Equation 2), and both were candidate fixes for the public-NLP-dataset regressions. The paper tested them against each other: raising the KL coefficient sacrifices substantial validation reward and never fully recovers performance on DROP and SQuADv2, while a well-chosen pretraining-mix coefficient reverses those regressions at minimal reward cost (instructgpt, §"4.2 Results on public NLP datasets", p. 15). The asymmetry is instructive about what each term actually does. The KL penalty constrains where the policy can go relative to the SFT model, so buying back capabilities with it means giving up alignment gains proportionally. Pretraining gradients instead inject the missing training signal directly — the lost capabilities live in the pretraining distribution, so restoring them requires that distribution, not just less movement. Notably, ptx does not make the KL knob redundant: even with the pretraining mix, β must still be tuned, with human Likert scores peaking near 0.01-0.02 and degrading at both 0 and 2 (instructgpt, §"E.7 Optimal KL reward coefficient", p. 56).