InstructGPT
Training language models to follow instructions with human feedback — introduced; Constitutional AI: Harmlessness from AI Feedback — inherited
The family of GPT-3 models fine-tuned with the paper's three-step demonstration-plus-RLHF pipeline, referred to by default as the PPO-ptx model unless otherwise specified.
InstructGPT is what GPT-3 became after OpenAI taught it to follow instructions, through human demonstrations, human preference judgments, and reinforcement learning arranged as a three-step pipeline. Its 2022 paper found people preferred the tuned model's outputs even over those of a much larger untuned one.
The page moves from what InstructGPT is to the headline comparison showing its gains came from data rather than scale, then to where it still falls short and how it went on to become the reference RLHF system downstream.
What InstructGPT is
InstructGPT is the paper's name for the family of GPT-3 models produced by its three-step fine-tuning pipeline, Supervised fine-tuning (SFT) on labeler demonstrations, reward-model training on labeler comparisons, and PPO fine-tuning stage (RLHF policy optimization) against that reward model. Unless otherwise specified, "InstructGPT" refers by default to the PPO-ptx variant, PPO-ptx (pretraining gradient mixing), which mixes pretraining-distribution gradients into the RL updates (instructgpt, §"3.5 Models", p. 9). Studied at three parameter scales, 1.3B, 6B, and 175B, all sharing GPT-3's architecture, InstructGPT is the paper's operational answer to its opening claim that the standard Language modeling (next-token prediction) objective is misaligned with what users actually want (instructgpt, Abstract, p. 1; §"1 Introduction", p. 2).
The headline comparison: data, not scale
The paper's central empirical claim, captured in the InstructGPT — outperforms at 100x smaller scale → GPT-3 (pretrained language model) edge, is that outputs from the 1.3B InstructGPT model are preferred to those of the 175B GPT-3 (pretrained language model) model despite having over 100x fewer parameters, and that 175B InstructGPT beats 175B GPT-3 head-to-head 85 ± 3% of the time (instructgpt, §"1 Introduction", p. 3). Because every InstructGPT model shares GPT-3's pretrained weights and architecture, the comparison isolates the effect of the fine-tuning pipeline itself: training the 175B PPO-ptx model cost about 60 petaflop/s-days against roughly 3,640 for pretraining GPT-3 (instructgpt, §"5.1 Implications for alignment research", p. 17), so a small fraction of pretraining compute, directed at human preference data via the InstructGPT's three-step training pipeline theme, outperformed a 100x increase in parameters on this task distribution.
Where it still falls short
InstructGPT's aggregate preference wins coexist with specific, documented failures collected under the Where RLHF alignment still falls short theme: it can still accept a Failure to detect false-premise instructions instruction as though its premise were true, produce Excessive hedging on simple questions on questions with a fairly clear answer, and exhibit residual Reward model over-optimization against the very reward model that trained it. The paper is explicit that these persist "despite its strong performance on many different language tasks" (instructgpt, §"4.3 Qualitative results", p. 16), qualifying InstructGPT's headline result rather than undermining it: RLHF works on this distribution, but not without residue.
Downstream footprint: becoming the reference RLHF system
Within this corpus, InstructGPT is also the concrete precedent that Constitutional AI positions itself against and partly reuses: Constitutional AI cites InstructGPT as related RLHF-based alignment work and, in Appendix D.3, draws comparison prompts directly from it (constitutional-ai, §"5 Related Work", p. 14; §"D.3 InstructGPT Prompts", p. 27). InstructGPT is thus not only this paper's endpoint but a fixed point of reference for the next paper in the corpus, which inherits its RLHF machinery while replacing the human labeler with an AI feedback model for harmlessness, a lineage traced by the From an open problem to reward models to AI feedback theme.