Web labeling interface
Training language models to follow instructions with human feedback — introduced
The internal web tool used by labelers and researchers to assign Likert quality scores, metadata labels, and full rankings to sets of model outputs for a given prompt.
Software choices that look incidental, like how a ranking task is laid out on screen, end up baked into a model's training data. This page is about the internal tool InstructGPT's labelers used to score and rank outputs, and what its design quietly determined.
The tool's place inside the paper's data pipeline is established first, then how its design shaped the data it produced, before the thread runs to the comparison dataset it feeds.
How it fits
The web labeling interface is the internal tool labelers and researchers actually use to assign Likert quality scores, metadata labels, and full rankings to sets of model outputs for a given prompt, part of Building the human side of the RLHF pipeline. It is introduced in this paper as purpose-built infrastructure rather than an off-the-shelf annotation tool (instructgpt, §"B.5 Web interface", p. 38).
What it quietly determines about the data
The interface's design choices shape the RM (comparison) dataset more than any written labeling instruction does: it filters uncertain comparisons out of the RM dataset through one specific affordance. It collects full rankings of $K=4$ to $9$ completions per prompt rather than isolated pairs, which is why the RM dataset's comparison count runs an order of magnitude beyond its roughly 33,000 prompts — each ranking expands into $\binom{K}{2}$ pairs via K-choose-2 single-batch comparison training. It also explicitly encourages ties "in cases where two outputs seem to be of similar quality" (instructgpt, §"B.5 Web interface", p. 39), and RM training then drops tied pairs entirely (instructgpt, §"C.2 Details of RM training", p. 42). Together these turn the interface's tie button into a quiet confidence filter: whenever a labeler cannot distinguish two outputs, that non-judgment is prevented from ever becoming gradient signal for the reward model, rather than being forced into an arbitrary label that would inject noise into the comparison loss. The dataset the reward model ultimately learns from is therefore not "everything labelers thought" but the subset of judgments they were confident enough to render decisively — a filtering effect built into UI design rather than into post-hoc data cleaning.