Reward model pairwise comparison loss
Training language models to follow instructions with human feedback — inherited
A cross-entropy loss over pairs of ranked completions in which the reward difference between preferred and dispreferred outputs represents the log odds a human would prefer the former.
When a labeler ranks two model outputs, that preference becomes training signal through a pairwise loss: the gap between the rewards assigned to the preferred and dispreferred outputs is read as the log odds a human would favor the first. InstructGPT (2022) inherited the loss from earlier summarization work.
What follows explains how the loss fits into the pipeline, walks through its mathematics, and draws out what its structure implies.
How it fits
This loss function is what actually trains the Reward model (RM), the proxy that makes InstructGPT's whole three-step pipeline tractable. The loss is not InstructGPT's invention: it is inherited, attributed directly to the comparison-based reward-model training approach of Stiennon et al. (2020)'s earlier RLHF summarization work, and it predates InstructGPT as a technique for turning pairwise human comparisons into a scalar reward signal (instructgpt, §"3.5 Models", p. 8). What is specific to InstructGPT is the engineering built around it: labelers rank $K=4$ to $9$ outputs per prompt rather than judging isolated pairs, and instead of shuffling every resulting comparison into an independent training example, InstructGPT places all comparisons from one ranking into a single batch element via K-choose-2 single-batch comparison training, which fixed an overfitting problem the naive shuffled scheme produced.
The mathematics
For a prompt $x$ and a pair of completions, let $y_w$ denote the completion a labeler preferred and $y_l$ the one they rejected. The reward model $r_\theta$ maps a prompt-completion pair to a scalar, and training minimizes
$$\text{loss}(\theta) = -\frac{1}{\binom{K}{2}} \, \mathbb{E}_{(x,\,y_w,\,y_l)\sim D}\Big[\log\big(\sigma(r_\theta(x,y_w) - r_\theta(x,y_l))\big)\Big]$$
where $D$ is the dataset of human comparisons, $\sigma$ is the logistic sigmoid, and the $1/\binom{K}{2}$ term averages over all pairwise comparisons drawn from one $K$-way ranking (instructgpt, §"3.5 Models", p. 8, Equation 1). The reward difference $r_\theta(x,y_w) - r_\theta(x,y_l)$ is treated as the log odds that a human would prefer $y_w$ to $y_l$: this is exactly a Bradley-Terry pairwise comparison model fit by logistic (cross-entropy) regression, with the reward model's scalar output playing the role of the latent per-item strength.
What the loss's structure implies
Because the loss depends only on differences $r_\theta(x,y_w) - r_\theta(x,y_l)$, adding any constant to every reward for a given $x$ leaves the loss completely unchanged — a mathematical fact with a real consequence documented in Reward model pairwise comparison loss — under determines the scale of → Reward model (RM): the trained ranker's absolute output is meaningless until an external anchor is imposed by convention. InstructGPT supplies that anchor after training, not through the loss, by shifting the reward model so labeler demonstrations score a mean of 0 (instructgpt, §"3.5 Models", p. 9). The pairwise formulation also explains a design choice one step upstream: because Equation 1 can only ever extract a preference probability from two items, there was no way to collect anything but comparisons in the first place, rather than, say, absolute quality scores.