Reward Prediction Normalization — prefigures the scale fix in → Reward model pairwise comparison loss

A comparison-only training objective leaves a reward predictor's absolute scale undetermined — nothing in the 2017 paper's loss changes if a constant is added to every predicted reward — so the paper rescales its predictor's output to zero mean and a fixed standard deviation (1 for MuJoCo, 0.05 for Atari) before handing it to the RL algorithm, a choice made so existing RL hyperparameters could be reused unchanged (deep-rl-human-prefs, §"2.2.1 Optimizing the Policy", p. 4; §"A.1 Simulated Robotics Tasks", p. 14). InstructGPT's reward model loss inherits exactly the same underdetermination for the same mathematical reason: a pairwise cross-entropy loss depends only on the difference between the two rewards being compared, so training on comparisons alone can never pin down where zero sits (instructgpt, §"3.5 Models", p. 8). The fix on the 2022 side is a renormalization too, but anchored differently: rather than picking a standard deviation for optimizer convenience, InstructGPT adds a bias so that labeler demonstrations score a mean of zero, making the anchor point interpretable — positive reward means better than a human demonstrator — rather than merely numerically convenient (instructgpt, §"3.5 Models", p. 9). The same defect, that comparison training cannot fix an absolute scale, gets patched twice six years apart: once for optimizer compatibility, once for interpretability.