Web labeling interface — filters uncertain comparisons out of → RM (comparison) dataset

Two interface decisions shape the RM dataset more than any labeling instruction. First, the interface collects full rankings of K = 4 to 9 completions per prompt rather than isolated pairs, which is why the dataset's comparison count runs an order of magnitude beyond its 33k prompts — each ranking expands into C(K,2) pairs. Second, the interface explicitly encourages ties "in cases where two outputs seem to be of similar quality" (instructgpt, §"B.5 Web interface", p. 39), and RM training then drops tied pairs entirely (instructgpt, §"C.2 Details of RM training", p. 42). Together these make the tie button a quiet confidence filter: whenever a labeler cannot distinguish two outputs, that non-judgment is prevented from ever becoming gradient signal, rather than being forced into an arbitrary 50/50 label that would add noise to the comparison loss. The dataset the reward model sees is therefore not "what labelers thought" but the subset of their judgments they were sure about, pre-cleaned by UI affordance rather than by post-hoc filtering.