Comparisons as the atomic unit of human feedback — assumes the error rate later measured in → Building the human side of the RLHF pipeline

The Bradley-Terry machinery this theme describes bakes in an assumption about its own labelers that it never checks: the constant rater-error noise model adds a flat 10% chance of a uniformly random response to the softmax, because "human raters have a constant probability of making an error, which doesn't decay to 0 as the difference in reward difference becomes extreme" (deep-rl-human-prefs, §"2.2.3 Fitting the Reward Function", p. 5). That figure is chosen once, at design time, and never measured against the paper's actual contractors. The human-feedback-data-infrastructure theme's inter-labeler agreement rate asks the same question the opposite way: instead of assuming a fixed error rate, InstructGPT's labeling operation measures directly, from real labeling data, how often independent labelers actually agree with each other (instructgpt, §"3.4 Human data collection", p. 8). The preference itself, one clip or completion judged against another, is the one thing that stays identical across five years; what changes is whether its unreliability is a constant guessed once or a statistic tracked as an ongoing property of the labeling workforce. Only the second version supports the held-out labeler generalization test, a check the 2017 paper's single hardcoded constant has no way to perform.