Instruction following (as a fine-tuning objective)
Training language models to follow instructions with human feedback — refined
The capability of a language model to carry out natural-language task instructions across an open-ended range of tasks, which this paper trains and measures directly via RLHF on real user-submitted prompts rather than via a fixed compilation of formatted NLP tasks.
Making a language model bigger does not automatically make it better at doing what you ask. InstructGPT treats that gap as the real problem, training and testing instruction-following directly on prompts real users submitted rather than a fixed set of formatted exercises.
Why the paper stakes its founding claim on this idea comes before how that claim gets tested through a ladder of model comparisons, then the thread runs out to the older next-token objective it replaces.
How it fits — the paper's founding methodological claim
Instruction following is InstructGPT's central thesis about what training objective a useful language model actually needs. The paper's opening argument is that the standard pretraining objective, predicting the next token on internet text, and what users actually want are different targets, and that scale alone does not close the gap: "making language models bigger does not inherently make them better at following a user's intent" (instructgpt, Abstract, p. 1). This is stated as a direct critique: the language modeling objective is argued to be misaligned with instruction following, since next-token prediction rewards plausible continuation of internet text rather than service to the user, and the paper states outright that "the language modeling objective is misaligned" (instructgpt, §"1 Introduction", p. 2). Instruction following is refined, not invented, as a training objective here — it draws on and is measured against two prior cross-task instruction-tuning efforts, FLAN and T0 — but InstructGPT trains and evaluates it in a structurally different way: directly, via RLHF on real user-submitted API prompts, rather than via a fixed compilation of reformatted NLP tasks.
What it enabled: a ladder of comparisons
The paper argues for this reframing empirically rather than by assertion, staging three separate comparisons that each isolate a different reason prior approaches fall short. FLAN sets the comparison floor: 175B InstructGPT is preferred over FLAN 78% of the time, and the paper attributes the gap to FLAN's fixed task compilation systematically underrepresenting real usage — classification and QA, easy to score automatically, are only about 18% of actual API traffic, while open-ended generation and brainstorming, poorly represented in FLAN, make up 57% (instructgpt, §"4.1 Results on the API distribution", p. 13). The GPT-3-prompted baseline shows the limits of prompting alone: even the best hand-crafted few-shot prefix, selected in an internal "prefix-finding competition" against the same reward model used for RLHF, still trails trained instruction following substantially, with 175B InstructGPT preferred 71±4% of the time. Together with a parallel comparison against T0, these results are what the win-rate metric was built to adjudicate — instruction-following's founding claim rests on measurement, not on the objective's definition alone.