Contractor Preference-Labeling Protocol
Deep Reinforcement Learning from Human Preferences — introduced
The paper's protocol for collecting real (non-synthetic) preference data at scale: paid contractors sign up for time slots, receive a 1-2 sentence task description (and, for Atari, first play the game themselves for five minutes), then repeatedly compare pairs of video clips via a keyboard-driven interface offering 'prefer left,' 'prefer right,' 'tie,' and 'can't tell' responses.
Behind every comparison used to train the reward model is a paid person clicking through an interface, and how they were recruited, instructed, and rushed shapes the results as much as the algorithm does.
The page covers the sign-up process and the interface itself, then converts a label count into hours and dollars of human time, before turning to places the protocol shows its seams, an implicit preference stated outright in instructions, an irregular result traced to one contractor's schedule, and closes on what later pipelines added that this one lacked.
The protocol itself
For every task requiring real, non-synthetic preference data, the paper recruits paid contractors rather than relying on the authors or open crowdsourcing. A contractor signs up "for a slot in the spreadsheet," is given "a 1-2 sentence description of each task" (deep-rl-human-prefs, §"3.1 Reinforcement Learning Tasks with Unobserved Rewards", p. 6), and — for Atari — first plays the game "for a five minutes to get a sense of how it works" (deep-rl-human-prefs, §"B.2 Atari", p. 16). Labeling then happens through a keyboard-driven clip-comparison interface: "left and right select clips, up is a tie, down is 'can't tell'" (deep-rl-human-prefs, §"B.1 MuJoCo", p. 16), giving contractors four possible responses to any pair of clips.
What the protocol is worth in labor terms
The Human-Feedback Sample Efficiency edge shows how this protocol converts an abstract label count into a labor cost: contractors "responded to the average query in 3-5 seconds, and so the experiments involving real human feedback required between 30 minutes and 5 hours of human time" (deep-rl-human-prefs, §"3.1 Reinforcement Learning Tasks with Unobserved Rewards", p. 6) — the figure grounding the abstract's "about an hour of human time" claim.
Where the protocol shows its seams
Several task-specific results expose the protocol's texture rather than its ideal case. On Ant (MuJoCo task), instructions made a supposedly "implicit" preference explicit: "the first priority is for the robot to be standing upright, and failing that for the center of the robot to be as high up as possible," with forward motion demoted to a tiebreaker (deep-rl-human-prefs, §"B.1 MuJoCo", p. 16). On Hopper (MuJoCo task), the irregular Figure 2 curve is attributed to labeling logistics, not the algorithm: "the irregular progress on Hopper is due to one contractor deviating from the typical labeling schedule" (deep-rl-human-prefs, §"3.1.1 Simulated Robotics", p. 7). Not every result used contractors: for Half-Cheetah (MuJoCo task) and Reacher, "feedback was provided by an author due to time constraints" (deep-rl-human-prefs, §"3.1.1 Simulated Robotics", p. 7). The label-annealing schedule is followed exactly only under Synthetic Oracle Feedback; with contractors it is approximate, since "contractors give feedback at uneven rates" (deep-rl-human-prefs, §"A Experimental Details", p. 14).
What it becomes downstream
The 2017 pipeline recruits labelers with no formal quality check: contractors sign up in a spreadsheet and begin labeling with no test of whether they are good at the task (deep-rl-human-prefs, §"B Instructions Provided to Contractors", p. 15). The labeler screening and selection process edge (hindsight) marks where later pipelines add a step this one lacks — a formal screen before labels count. The web labeling interface edge (hindsight) marks how the ad hoc clip server and spreadsheet are formalized into a dedicated product, though the two differ enough that the relation is a loose lineage, not a direct correspondence.