The training objective is a small system of knobs, not one number
Call it a reward function and you'd miss what is actually driving InstructGPT's PPO stage: a KL penalty doing double duty, discouraging drift and, almost as an accident, supplying the only dense signal in an otherwise sparse reward.
From there the page separates that penalty's two jobs, then turns to the bandit setup that makes discounting a non-issue in the first place. A distinct third knob, the pretraining-gradient mix, gets its own treatment: proof that capability can be recovered without spending down what RLHF gained.
InstructGPT's PPO objective is a system of interacting design choices rather than a single reward. Reward overoptimization is preemptively countered by the KL penalty, applied per token specifically so the counter-pressure tracks the same surface reward-hacking could exploit. In the single-step bandit environment that RLHF actually trains on, the bandit setting renders discounting moot in generalized advantage estimation, since there is no future left to down-weight, while the KL penalty simultaneously densifies the terminal reward of that same bandit environment, becoming the objective's only dense per-token signal. The paper keeps this term strictly separate from a second one: reward overoptimization demands a different mitigation than the alignment tax, and PPO-ptx outperforms simply tightening the KL penalty at fixing public-NLP-dataset regressions, while diverging from plain PPO only off-distribution, which is what lets the paper show the tax can be paid down without touching the alignment gain.