Reward model pairwise comparison loss — under determines the scale of → Reward model (RM)

The pairwise cross-entropy loss sees only differences between rewards for preferred and dispreferred completions, so adding any constant to every reward leaves the loss unchanged: training produces a ranker whose absolute output is meaningless. This matters the moment the ranker is reused as an RL reward, where absolute values feed the optimization directly. The paper handles it with an explicit post-hoc step, adding a bias so that labeler demonstrations score a mean of 0 before RL begins (instructgpt, §"3.5 Models", p. 9). The choice of anchor is quietly substantive: zero is defined as "what a human demonstrator would have written," so positive reward literally means better-than-demonstration, and PPO's objective becomes exceeding the humans who taught the SFT stage. The loss also explains why comparisons rather than scores were collected in the first place — a preference probability is all Equation 1 can extract — and why calibration of the reward scale had to come from outside the training signal entirely.