Instruction following as an alternative to next-token prediction
Predicting the next word of internet text and doing what someone actually asks are not the same skill, and InstructGPT (2022) treats that gap as the whole point. This theme covers the argument that instruction-following deserves to be its own training objective, made by comparing it against its nearest rivals: instruction-tuned compilations like FLAN and T0, and a plainly prompted GPT-3.
The comparison runs in full against all three rivals, with the language-modeling objective itself sitting in the background throughout, the baseline everything else in this theme is defined against.
InstructGPT argues that the standard language-modeling objective, predicting the next token on internet text, is misaligned with what users actually want, which is a model that helpfully and safely follows instructions across an open-ended range of tasks. This theme gathers the concepts that frame that argument. FLAN and T0 are two prior cross-task instruction-tuning baselines, each a compilation of NLP tasks reformatted with natural-language instructions and used to fine-tune a 175B GPT-3 model for comparison. The GPT-3 prompted baseline instead tries to coax instruction-following behavior out of an unmodified model with a hand-crafted few-shot prefix. Instruction following itself is defined and measured here directly, via RLHF trained on real user-submitted prompts, in explicit contrast to both the fixed-compilation approach of FLAN and T0 and the prompting-only approach of the GPT-3 baseline. The claim uniting these concepts is methodological: instruction-following is not one task among many but a different training objective from next-token prediction, argued for by comparison against its nearest rivals.