Constant Rater-Error Noise Model
Deep Reinforcement Learning from Human Preferences — introduced
A modification to the preference-prediction softmax assuming a constant 10% chance the human responds uniformly at random, modeling the fact that human raters have a non-vanishing error rate rather than becoming perfectly accurate as the true reward difference grows extreme.
A plain preference model implies that people grow perfectly confident once two options are different enough. That isn't how actual human judgment works, and the paper builds in an assumption to correct for it.
The page covers the fixed error rate assumed regardless of how large a reward gap gets, then the reasoning behind it, before closing on how a later paper eventually measures, rather than assumes, roughly the same underlying question about how much to trust a single human judgment.
A floor on human accuracy
Applied directly, the Bradley-Terry softmax in Equation 1 predicts that a human's probability of preferring the higher-reward segment approaches 1 as the gap between the two summed predicted rewards grows arbitrarily large. The paper rejects that asymptotic behavior by construction: rather than applying the softmax directly, it assumes there is a constant 10% chance the human responds uniformly at random, regardless of how large the reward gap is (deep-rl-human-prefs, §"2.2.3 Fitting the Reward Function", p. 5). The stated reason is that "human raters have a constant probability of making an error, which doesn't decay to 0 as the difference in reward difference becomes extreme" (deep-rl-human-prefs, §"2.2.3 Fitting the Reward Function", p. 5).
An assumption, not yet a measurement
The 10% figure is asserted rather than estimated from any data the 2017 experiments collect; nothing in this paper measures how often two raters actually disagree on the same pair. That measurement appears only later: InstructGPT reports the rate directly, finding that training labelers agree with each other 72.6 ± 1.5% of the time and held-out labelers agree 77.3 ± 1.3% of the time (instructgpt, §"3.4 Human data collection", p. 8). The inter-labeler agreement rate gives an empirical number to set against the constant-error assumption built in here by hand; the two are not the same quantity — one is a modeling assumption inside a loss function, the other a measured property of a labeling workforce — but they answer the same underlying question about how much to trust a single human judgment.