Held-out labeler generalization test
Training language models to follow instructions with human feedback — introduced
An experiment measuring whether models and reward models trained on one group of labelers' preferences generalize to a separate group of labelers who contributed no training data, via both a held-out labeler evaluation and 5-fold cross-validated reward models.
Results built on one small group's preferences might just reflect that group's particular tastes rather than anything more broadly shared. InstructGPT tests for this directly, checking whether its results hold up for labelers who never touched the training data.
Where this test sits in the paper's larger argument is explained first, then why it mattered and what it found, before looping back to the screening process that assembled the original labeler pool.
How it fits
This experiment is InstructGPT's check on whether its headline results reflect a broad human consensus or merely the tastes of a small, curated group. It sits inside Building the human side of the RLHF pipeline as the concept that tests whether the whole human-data pipeline generalizes, complementing Inter-labeler agreement rate, which tests whether labelers agree with each other at all. It is introduced by this paper, combining a held-out labeler evaluation with 5-fold cross-validated reward models (instructgpt, §"3.4 Human data collection", p. 8; §"E.2 Reward model generalization across sets of labelers", p. 52).
Why it was necessary and what it found
The labeler screening process is what makes this test necessary in the first place. Every training label came from roughly 40 contractors filtered through four criteria, including agreement with the researchers' own judgments (instructgpt, §"B.1 Labeler selection", p. 36), so the headline preference results could in principle reflect a small, curated, researcher-aligned group rather than anything broader. The held-out experiment breaks that dependence directly: the additional labelers are sourced from the same vendors but "do not undergo a screening test" (instructgpt, §"3.4 Human data collection", p. 8), so passing the screening filter cannot explain their judgments. The result: held-out labelers prefer InstructGPT to GPT-3 at rates similar to the training labelers, and five-fold cross-validated reward models predict held-out labeler groups' preferences at 69.6% accuracy versus 72.4% for in-group prediction (instructgpt, §"E.2 Reward model generalization across sets of labelers", p. 52) — a real but modest overfitting-to-labelers cost. The test answers the specific question the screening process created; whether roughly 40 contractors represent the broader population of eventual users is a separate, harder question this test does not address.