Inter-labeler agreement rate
Training language models to follow instructions with human feedback — inherited
The measured rate at which independent labelers agree with each other (or with researchers) when ranking or labeling the same outputs, used to validate the feasibility and consistency of the labeling task.
If two people reading the same model output cannot agree on which is better, no downstream number built from their judgments means much either. Inter-labeler agreement measures exactly that overlap, and it quietly sets a ceiling on how good any reported win rate can look.
It traces how this single measurement pulls double duty, filtering candidate labelers before they start and auditing the data as it comes in, then follows the thread to the screening process it feeds and the win-rate figures it bounds.
Measuring agreement, in general
Inter-annotator (or inter-labeler) agreement is a standard measurement in any human-labeling pipeline, quantifying how consistently independent annotators assign the same label or ranking to the same item. The simplest form, and the one this paper reports, is raw percent agreement, the fraction of items on which two annotators agree, as opposed to chance-corrected measures such as Cohen's or Fleiss' kappa that subtract out the agreement expected from random guessing. Percent agreement predates this paper as a basic tool of annotation methodology; InstructGPT inherits it as its chosen way to validate that its ranking task is well-posed (instructgpt, §"3.4 Human data collection", p. 8).
Its double role: filter and audit
Agreement enters InstructGPT's pipeline on both sides of a causal arrow, as the Inter-labeler agreement rate — was both filter and audit for → Labeler screening and selection process edge details. During selection, two of the four Labeler screening and selection process criteria are agreement measures against researcher labels, with soft cutoffs at 75% (instructgpt, §"B.1 Labeler selection", p. 36). After training, agreement is reported as evidence the labeling task itself is tractable: training labelers agree with each other 72.6 ± 1.5% of the time, comparable to the 73 ± 4% researcher-researcher agreement Stiennon et al. (2020) reported for the narrower task of summarization (instructgpt, §"3.4 Human data collection", p. 8). Notably, the unscreened held-out labelers agree among themselves more, 77.3 ± 1.3%, than the screened training group does, suggesting the screening criteria selected for sensitivity to harmful content across demographic groups rather than for uniformity of taste.
The floor it puts under every win rate
A Win rate (preference against baseline) is a proportion of human choices, so it cannot be more precise than the humans making those choices are internally consistent; the Inter-labeler agreement rate — caps the resolution of → Win rate (preference against baseline) edge makes this explicit. With roughly a quarter of individual pairwise comparisons behaving like coin flips, the paper's headline 85% preference figure for 175B InstructGPT over GPT-3 (instructgpt, §"4.1 Results on the API distribution", p. 11) towers safely over that noise floor, but smaller gaps between adjacent models, such as PPO (without pretraining mix) versus PPO-ptx (pretraining gradient mixing), sit inside it and should not be read as firm rankings. This concept belongs to both the Building the human side of the RLHF pipeline theme, as a validity check on the labeling pipeline, and the Automatic overlap scores versus human and preference judgments theme, as the consistency floor underlying every judgment-based metric in the corpus.