What comparison training leaves undetermined
Learn a reward purely from which of two things a person preferred, and the result is pinned down only up to reordering: nothing fixes its zero point, scale, or how confident a prediction may get. The Bradley-Terry model supplies the comparison loss, but its bare softmax would let confidence climb toward certainty, so a constant rater-error rate caps it, and normalization fixes a scale the comparisons never specified.
The walk works through each patch, including reward being summed across a clip without discounting, closing on two smaller facts: a normalization constant chosen backward from an algorithm's existing hyperparameters, and the same scale problem reappearing, patched differently, in InstructGPT years later.
A reward model learned only from comparisons is pinned down only up to order-preserving transformations, so these edges are the patches that nail down what pairwise ranking alone leaves free: the loss function itself, the asymptotic confidence of that loss, the model's absolute scale, and the timing assumptions within a single clip. The Bradley-Terry model supplies the loss for the reward model, fitting predicted summed reward as a latent score inside a pairwise-comparison softmax rather than regressing to any absolute number, since comparisons are all the training data provides. The rater error noise model corrects an asymptotic property a bare Bradley-Terry softmax gets wrong, capping predicted confidence well below certainty with an assumed constant 10 percent error rate rather than letting confidence approach 1 as the reward gap between two segments grows. Reward normalization fixes the underdetermined scale of the reward model, rescaling its output to zero mean and a fixed standard deviation before the RL algorithm sees it, since the comparison loss alone leaves both the zero point and the magnitude unconstrained. The trajectory segment is summed without discount by the Bradley-Terry model, an explicit choice that treats an event at the start of a one-to-two-second clip as equally decisive as the same event at the end, one more free parameter the comparison framing has to fix by assumption rather than infer from the data. Reward normalization is reverse-engineered to avoid retuning A2C: the fixed standard deviation of 0.05 chosen for Atari was picked backward from A2C's existing hyperparameters rather than from any principled measure of scale. And reward normalization prefigures the scale fix that reappears in the reward model loss five years later, when InstructGPT anchors its own zero point to labeler demonstrations instead, patching the identical underdetermination for interpretability rather than for optimizer convenience.