Reward Prediction Normalization
Deep Reinforcement Learning from Human Preferences — introduced
A preprocessing step in which the reward predictor's output is rescaled to have zero mean and a fixed standard deviation before being used by the RL algorithm, since the absolute scale and offset of a learned reward function is otherwise underdetermined by the comparison-based training objective.
A reward model trained only on comparisons never sees an absolute number, which leaves its output's scale and zero point free to drift wherever training happens to put them.
The page covers how the paper pins that scale down before handing the reward model's output to the RL algorithm, then why the specific number chosen for Atari was picked to avoid retuning anything else, and closes on how a later system solves the identical problem inside its loss function instead of as a separate step.
An underdetermined scale, pinned down
A reward predictor trained only on comparisons never sees an absolute number — nothing in the Bradley-Terry loss changes if a constant is added to every predicted reward, since only differences between two segments' summed rewards enter the softmax. The paper closes off this free parameter by rescaling the output of $\hat r$ to zero mean and a fixed standard deviation before it reaches the RL algorithm, calling this "a typical preprocessing step which is particularly appropriate here since the position of the rewards is underdetermined by our learning problem" (deep-rl-human-prefs, §"2.2.1 Optimizing the Policy", p. 5).
Choosing the standard deviation to avoid retuning A2C
The fixed standard deviation is not the same across domains: it is 1 for the MuJoCo tasks (deep-rl-human-prefs, §"A.1 Simulated Robotics Tasks", p. 14) and 0.05 for Atari (deep-rl-human-prefs, §"A.2 Atari", p. 15). The paper calls the Atari figure arbitrary in principle but chosen specifically so the same A2C hyperparameters used against the true reward — entropy bonus, learning rate — could be reused unchanged, rather than retuned for a learned reward's different scale (deep-rl-human-prefs, §"A.2 Atari", p. 15).
A fix later folded into the loss itself
This paper solves the scale problem as a preprocessing step external to the loss: normalize, then hand the result to an unmodified RL algorithm. InstructGPT's reward-model loss folds an equivalent fix into training instead — its sigmoid-of-a-difference form is scale-invariant by construction, so no separate renormalization step is needed before optimization. The underlying problem is the one this concept names first: a comparison-only objective leaves a reward model's absolute scale undetermined, and something downstream has to pin it down, whether by explicit renormalization or by a loss whose gradient never depended on the absolute scale to begin with. The concept sits at the intersection of the Keeping the reward model on-distribution and Policy optimization under a learned, changing reward themes.