Win rate (preference against baseline)
Training language models to follow instructions with human feedback — inherited
The primary human-evaluation metric: the fraction of times labelers prefer one model's output over that of a fixed baseline policy (the 175B SFT model) on matched prompts.
The headline results of InstructGPT come down to a simple count: shown two responses to the same prompt, how often do labelers pick this model's over the baseline's? That fraction, measured against the 175B supervised model, is the paper's primary measure of quality.
What bounds this number, what confirms it, and what it quietly misses is worth more attention than the metric's mechanics. Likert ratings and inter-labeler agreement, both connected here, round out a picture no single fraction can give.
How it fits
Win rate is InstructGPT's primary human-evaluation metric and, within Automatic overlap scores versus human and preference judgments, the central instrument the corpus uses to say quantitatively that one model is preferred to another: the fraction of times labelers prefer one model's output over a fixed baseline policy, the 175B SFT model, on matched prompts. It is inherited rather than invented here — pairwise preference-rate evaluation predates this paper — and the headline result it produces, 175B InstructGPT preferred to 175B GPT-3 85% of the time and to few-shot GPT-3 71% of the time, is the paper's single most-cited number (instructgpt, §"3.6 Evaluation", p. 10; §"4.1 Results on the API distribution", p. 11).
What bounds it, what confirms it, and what it misses
Win rate's precision has a hard floor set by the humans producing it: inter-annotator agreement caps its resolution, since training labelers agree with each other only 72.6±1.5% of the time and held-out labelers 77.3±1.3% (instructgpt, §"3.4 Human data collection", p. 8) — meaning roughly a quarter of individual comparisons are effectively coin flips, and small gaps between adjacent models should not be read as reliable rankings even though the 85% headline result comfortably clears that noise floor. Two further instruments extend win rate rather than merely repeating it. The Likert quality rating corroborates it on an anchor-free scale, confirming the ordering is not an artifact of the chosen baseline. And in Constitutional AI, the Elo score generalizes win rate to handle 24 model snapshots across multiple training runs at once, something a single-anchor win rate cannot do, at the cost of an arbitrary zero point Elo must separately calibrate (constitutional-ai, §"3.3 Main Results", p. 8; §"4.3 Main Results", p. 12). Win rate's most consequential limitation surfaces against a very different kind of instrument: it diverges sharply under RLHF from F1 score. PPO fine-tuning pushes win rate to its 85% headline while collapsing F1 on extractive QA benchmarks such as DROP and SQuAD v2 (instructgpt, §"E.1 Performance on public NLP datasets", Table 14, p. 56). InstructGPT resolves the disagreement asymmetrically, treating win rate as the actual training objective and F1 as a constraint to be satisfied afterward — a methodological choice that makes win rate the metric the entire pipeline is ultimately built to optimize, and everything else a check against what it cannot see.