Comparisons as the atomic unit of human feedback — scales binary comparison into rankings for → InstructGPT's three-step training pipeline

The 2017 mechanism only ever resolves one pair at a time: a human is shown two trajectory segments, and the Bradley-Terry softmax converts that single binary judgment into a scalar reward via "the difference in predicted reward of two trajectory segments" (deep-rl-human-prefs, §"2.2.3 Fitting the Reward Function", p. 5). InstructGPT's reward-model step generalizes the same softmax to whole rankings: labelers rank K = 4 to 9 outputs per prompt rather than choosing between two (instructgpt, §"3.2 Dataset", p. 7), and the reward model pairwise comparison loss is applied to every pair implied by that ranking at once. Doing so creates a problem 2017 never faces, because each of its comparisons is drawn as an independently sampled pair; comparisons pulled from one ranking are correlated, sharing K-1 outputs with each other, so training on all of them as if they were independently shuffled data overfits. K-choose-2 batching is the specific fix: every pairwise comparison from one labeler's ranked set is placed into a single batch element together, rather than scattered through training as if unrelated (instructgpt, §"3.5 Models", p. 8). Nothing about the underlying comparison-to-scale logic changes in this generalization from pairs to rankings, only the bookkeeping needed to keep correlated comparisons from being double-counted as independent evidence.