Preference Elicitation Protocol
Deep Reinforcement Learning from Human Preferences — introduced
The procedure by which a human overseer is shown two rendered trajectory-segment video clips and indicates a preference, a tie, or that the segments are incomparable, with responses recorded as a probability distribution over the two segments in a preference database.
Two clips, and three possible answers: better, equal, or too hard to say. Almost everything else in the method depends on this one small interaction happening correctly.
The page describes the interaction itself and how each answer gets recorded, then situates it against other ways a human's goal could have been communicated instead, before closing on the one constraint, a single comparison per query, that a later system built on the same recipe eventually let go of.
Two clips, three possible answers
The protocol shows a human overseer two trajectory-segment video clips and asks them to say which is better, that the two are equally good, or that they cannot be compared (deep-rl-human-prefs, §"2.2.2 Preference Elicitation", p. 5). Every judgment is recorded in a database $D$ as a triple $(\sigma^1,\sigma^2,\mu)$, where $\mu$ is a distribution over $\{1,2\}$: a strict preference puts all its mass on the chosen segment, a tie splits it uniformly, and an "incomparable" judgment is dropped from the database entirely rather than stored as a null preference (deep-rl-human-prefs, §"2.2.2 Preference Elicitation", p. 5). This database is what the Bradley-Terry Model loss is later minimized against.
Preferences instead of demonstrations or free interaction
The protocol is a specific, narrowed answer to a broader question about how a human should communicate a goal. Concrete Problems (2016) had proposed mining demonstrations via inverse RL or apprenticeship learning as a way to substitute human expertise for a hand-specified objective (concrete-problems, §"Use Demonstrations:", p. 15); the 2017 paper explicitly passes on this, since inverse RL and imitation learning "are not directly applicable to behaviors that are difficult for humans to demonstrate," such as controlling a robot with non-human morphology (deep-rl-human-prefs, §"1 Introduction", p. 1). It also positions itself as a special case of cooperative inverse reinforcement learning, the two-player game in which a human and robot cooperate to maximize the human's reward — but narrowed so "the human is only allowed to interact with this game by stating their preferences" (deep-rl-human-prefs, §"1.1 Related Work", p. 3), rather than acting freely to demonstrate what is wanted.
One pair at a time, and what that later cost
Every call to the protocol produces exactly one triple, independent of any other comparison the human has made. This independence is simple but, in hindsight, wasteful: InstructGPT's labelers instead rank K = 4 to 9 responses per prompt at once, producing (K choose 2) comparisons from a single ranking task. Training naively on those as independent points caused the reward model to overfit after a single pass, since comparisons drawn from the same ranking are highly correlated — a problem the K-choose-2 batching fix addresses by treating all (K choose 2) comparisons from one prompt as a single batch element (instructgpt, §"3.5 Models", p. 8). The 2017 protocol never encounters this failure mode, because it never asks for more than a single comparison per query in the first place.