Contractor Preference-Labeling Protocol — is formalized as → Web labeling interface

In 2017, the entire labeling apparatus is an ad hoc clip server plus a sign-up spreadsheet: a contractor signs up for a slot, receives a one-to-two sentence description of the task, and is then repeatedly shown two video clips to compare, selecting the one in which better things happen (deep-rl-human-prefs, §"B.2 Atari", p. 16). When the server ran out of clips, the contractor was told this was 'normal, just wait a minute and refresh the page,' and if the problem persisted, to 'ping @tom on slack' — infrastructure held together by one named person watching a chat channel (deep-rl-human-prefs, §"B.2 Atari", p. 17). InstructGPT's web labeling interface is the same basic function — show an item, collect a structured judgment — rebuilt as an actual internal product: labelers assign Likert quality scores, apply a fixed taxonomy of metadata labels, and produce full rankings over sets of outputs for a given prompt, all within one tool (instructgpt, §"B.5 Web interface", p. 38). The tie-or-can't-tell judgments the 2017 interface offered survive conceptually in 2022's low-confidence and quality-score fields, but they are now one signal among several captured per item, rather than the entirety of what the interface records.