Comparisons as the atomic unit of human feedback
Ask someone to score a robot's behavior on some absolute scale and their answers wobble; ask which of two clips is better and they agree with themselves far more often. The 2017 paper builds its entire reward-learning method on that one observation, a choice that resurfaces unchanged five years later inside InstructGPT's (2022) chatbots.
The walk opens on the trajectory segment being judged, then the protocol eliciting a preference on it and the Bradley-Terry math turning many judgments into a scale, tested against direct scoring and corrected for rater error. It closes on the Elo analogy explaining that scale and the reward model the process produces.
This theme's claim is that one design decision, asking a human to compare two outcomes rather than score one, is the load-bearing joint connecting the 2017 paper's robots to the chatbots of 2022. The trajectory segment is the unit being compared: a short clip of observations and actions, one to two seconds long, rather than a whole episode or a single state. The preference elicitation protocol is the procedure that shows a human two such clips and records a preference, a tie, or an incomparable judgment in a growing database. The Bradley-Terry model is the statistical machinery, borrowed from 1950s paired-comparison theory, that turns those recorded preferences into a scalar reward scale via a softmax over predicted per-segment reward; the same comparison-to-scale logic reappears, unnamed but unchanged, in the reward model loss that trains InstructGPT's (2022) reward model on ranked completions. The paper explains its own scale with an Elo score analogy, noting that a difference in predicted reward plays the same role as a difference in chess rating, the same rating concept Constitutional AI (2022) later computes literally to rank model snapshots. The comparisons vs. absolute scores ablation is the paper's direct test of the choice to elicit comparisons at all, motivated by the finding that humans give more consistent comparisons than consistent scores, though regressing directly to a score performs about as well on some tasks. Because raters are not perfectly accurate even on easy comparisons, the constant rater-error noise model adds a floor probability of a random response so the predicted preference distribution never asymptotes to certainty. The reward model is the artifact all of this produces: a network trained on comparisons that stands in for the missing reward function, the same canonical model InstructGPT (2022) later trains on human rankings of language outputs.