GPT-3 (pretrained language model)
Training language models to follow instructions with human feedback — inherited
The autoregressive pretrained language model from Brown et al. (2020) that serves as the starting point and architecture for all SFT, reward, and PPO models in this paper.
Underneath everything in the InstructGPT work sits GPT-3, the large autoregressive language model that Brown and colleagues introduced in 2020, trained to predict the next token across a vast sweep of text. Every model the paper trains, supervised, reward, or PPO, starts from it.
The prose explains what GPT-3 is, then its role in this corpus as the shared starting point, and finally a controlled comparison that teaches a lesson about where the improvements really came from.
What GPT-3 is
GPT-3 (Brown et al., 2020) is a large autoregressive, decoder-only Transformer language model trained to predict the next token of text given all preceding tokens. For a sequence $x_1, \ldots, x_T$, it is trained by maximum likelihood under the factorization $$P_\theta(x_1,\ldots,x_T) = \prod_{t=1}^{T} P_\theta(x_t \mid x_{<t}),$$ trained on a broad corpus of internet text at scales up to 175 billion parameters. Its signature property, demonstrated at the scale of the original paper, is strong few-shot in-context learning: presented with a handful of task examples as part of its input prompt, without any weight updates, GPT-3 could perform a wide range of NLP tasks reasonably well. GPT-3 predates InstructGPT by roughly a year and a half and is inherited wholesale here: it supplies both the architecture and the pretrained weights for every model in the paper, not a technique InstructGPT modifies (instructgpt, §"3.5 Models", p. 8).
Its role in this corpus: the shared starting point
Every InstructGPT model, at every one of the three studied sizes (1.3B, 6B, and 175B parameters), begins from GPT-3's pretrained weights and architecture before any of the paper's three fine-tuning steps are applied (instructgpt, §"3.5 Models", p. 8; §"C Additional model details", p. 40). GPT-3 is also the paper's central un-fine-tuned baseline, evaluated both unprompted and with a hand-crafted few-shot prefix (GPT-3-prompted) designed to coax it into an instruction-following mode (instructgpt, §"3.5 Models", p. 9). Because SFT, RM, and PPO models are all architecturally identical to GPT-3 and differ only in what data and objective trained them, GPT-3 functions in this paper less as a rival system than as the fixed control against which every intervention in the InstructGPT's three-step training pipeline theme is measured.
A controlled comparison, and a lesson about where the gains came from
The InstructGPT — outperforms at 100x smaller scale → GPT-3 (pretrained language model) edge turns this shared architecture into a genuine controlled experiment rather than a mere benchmark win: because InstructGPT and GPT-3 share weights and architecture up to the point of fine-tuning, any preference gap isolates the effect of the human-feedback pipeline. Labelers prefer outputs from the 1.3B InstructGPT model to those of the full 175B GPT-3, and 175B InstructGPT beats 175B GPT-3 head-to-head 85 ± 3% of the time (instructgpt, §"1 Introduction", p. 3). The cost asymmetry sharpens the point: training the 175B PPO-ptx model took about 60 petaflop/s-days against roughly 3,640 for pretraining GPT-3 itself (instructgpt, §"5.1 Implications for alignment research", p. 17), so under 2% of pretraining compute bought a larger preference gain than a 100x increase in parameter count. On this evidence, alignment data, not raw scale, was the binding lever for this distribution of tasks.