Bradley-Terry Model
Deep Reinforcement Learning from Human Preferences — inherited
A statistical model (Bradley and Terry, 1952) for estimating latent scores from pairwise comparison outcomes, used here to convert predicted per-segment summed reward into a probability that a human prefers one segment over another via a softmax over the two sums.
Comparing two chess players by the gap in their ratings is a much older idea than deep reinforcement learning, and it solves a real problem here too: how to turn a difference in predicted reward into a probability that a person prefers one clip over another.
The page introduces the model's pairwise-comparison math, then the chess analogy the paper uses to explain it, before covering two adjustments made to the plain formula and how the same loss reappears, essentially unchanged, five years later.
Turning a reward difference into a probability
The Bradley-Terry model (Bradley and Terry, 1952) estimates a latent score for each item in a set from the outcomes of pairwise comparisons; here the items are trajectory segments and the score is the reward predictor's own output. The paper interprets $\hat r$ as a preference predictor by assuming a human's probability of preferring segment $\sigma^1$ depends exponentially on predicted reward summed over the segment (deep-rl-human-prefs, §"2.2.3 Fitting the Reward Function", p. 5): $$\hat P[\sigma^1 \succ \sigma^2] = \frac{\exp \sum_t \hat r(o^1_t,a^1_t)}{\exp \sum_t \hat r(o^1_t,a^1_t) + \exp \sum_t \hat r(o^2_t,a^2_t)}$$ $\hat r$ is then chosen to minimize the cross-entropy loss between this prediction and the recorded label $\mu$: $$\text{loss}(\hat r) = -\sum_{(\sigma^1,\sigma^2,\mu)\in D} \mu(1)\log\hat P[\sigma^1\succ\sigma^2] + \mu(2)\log\hat P[\sigma^2\succ\sigma^1]$$ (deep-rl-human-prefs, §"2.2.3 Fitting the Reward Function", p. 5). The trajectory segments $\sigma^1$, $\sigma^2$ being compared are summed without a discount factor, a modeling choice the paper flags rather than treats as neutral.
A named model, not a bespoke invention
The paper is explicit this is not a new construction: it names the loss as "the Bradley-Terry model...for estimating score functions from pairwise preferences," and identifies it as the specialization of the Luce-Shepard choice rule to preferences over trajectory segments (deep-rl-human-prefs, §"2.2.3 Fitting the Reward Function", p. 5). It offers a chess analogy to make the mechanism intuitive: the Elo rating system estimates the probability that one player beats another from the difference in their ratings, and the same structure is used here — the difference in predicted reward between two segments estimates the probability a human prefers one to the other (deep-rl-human-prefs, §"2.2.3 Fitting the Reward Function", p. 5). No Elo computation is actually performed; the analogy is purely explanatory.
Two adjustments to the basic model
The paper modifies the plain softmax in two ways it separately analyzes with ablations. The Constant Rater-Error Noise Model rejects the softmax's implication that a human becomes arbitrarily accurate as the reward gap grows: it assumes instead a constant 10% chance of a uniformly random response, since human error "doesn't decay to 0" as the difference becomes extreme (deep-rl-human-prefs, §"2.2.3 Fitting the Reward Function", p. 5). Separately, the Comparisons vs. Absolute Scores (Target Ablation) ablation removes the comparison framing altogether, replacing cross-entropy over pairwise judgments with mean-squared-error regression to an oracle's true summed reward — a direct test of whether eliciting comparisons, rather than scores, is what makes the approach work (deep-rl-human-prefs, §"3.3 Ablation Studies", p. 9).
The same loss, five years later
InstructGPT's reward-model loss keeps this structure: predicted scalar rewards for two completions are passed through the same sigmoid-of-a-difference form and fit by cross-entropy over human comparisons (instructgpt, §"3.5 Models", p. 8). What changes downstream is the data-collection layer feeding it — K-choose-2 ranking batches instead of single pairwise queries — not the Bradley-Terry mathematics, which carries over essentially unmodified from Equation 1. The concept anchors the Comparisons as the atomic unit of human feedback theme running through both papers.