RM (comparison) dataset
Training language models to follow instructions with human feedback — introduced
The set of about 33k training prompts, each with K=4 to 9 model outputs ranked by labelers, used to train the reward model.
To learn what people prefer, you first need a record of people preferring things. This dataset is that record: around 33,000 prompts, each with a handful of model outputs that labelers ranked from best to worst, gathered to train InstructGPT's reward model.
It sits among the pipeline's three human-data artifacts, but the more consequential half of the page is what shapes the data before the reward model ever sees it, since bias introduced here becomes bias everywhere downstream.
How it fits
The RM dataset is the second of three human-data artifacts in InstructGPT's Building the human side of the RLHF pipeline, sitting between the SFT dataset that precedes it and the PPO dataset that follows: roughly 33,000 training prompts, each paired with $K=4$ to $9$ model outputs ranked by labelers, used to train the Reward model (RM) (instructgpt, §"3.2 Dataset", p. 7; §"A.3 Dataset sizes", p. 33). Its comparison count runs an order of magnitude beyond its prompt count because each ranking of $K$ outputs expands into $\binom{K}{2}$ pairwise comparisons via K-choose-2 single-batch comparison training.
What shapes it before the reward model ever sees it
Two upstream choices determine what the reward model actually learns from this dataset, neither of which is a labeling instruction. First, the completions being ranked come overwhelmingly from SFT-trained policies — "most of our comparison data comes from our supervised policies, with some coming from our PPO policies" (instructgpt, §"3.1 High-level methodology", p. 6) — so the reward model's competence is local to the neighborhood of output space SFT-style policies actually produce, not to the full space of possible completions. Second, the labeling interface itself filters out uncertain comparisons: it explicitly encourages ties "in cases where two outputs seem to be of similar quality" (instructgpt, §"B.5 Web interface", p. 39), and RM training then drops tied pairs entirely (instructgpt, §"C.2 Details of RM training", p. 42). The tie button functions as a built-in confidence filter — whenever a labeler cannot distinguish two outputs, that non-judgment never becomes gradient signal, rather than being forced into an arbitrary label that would add noise. The dataset the reward model actually trains on is therefore not "everything labelers judged" but the subset of judgments labelers were confident enough about to rank decisively.