Bradley-Terry Model — supplies the loss for → Reward model (RM)

The reward model has no target values to regress against; it is fit entirely by treating predicted summed reward as a latent score in the Bradley-Terry pairwise-comparison model and minimizing the resulting cross-entropy loss over recorded human preferences (deep-rl-human-prefs, §"2.2.3 Fitting the Reward Function", p. 5). Under this loss, a segment's predicted probability of being preferred is a softmax over the two segments' summed rewards, so what the network learns is not an absolute reward scale but a ranking consistent with observed preferences. The paper notes this is the same construction as the Luce-Shephard choice rule, a much older result about how proximity in an internal similarity or utility space predicts choice probabilities, applied here to segment rewards instead of psychological stimuli. Borrowing this specific loss, rather than direct regression, is what lets the reward model be trained purely from comparisons, without ever seeing a numeric reward target.