Language modeling (next-token prediction) objective
Training language models to follow instructions with human feedback — inherited
The standard pretraining objective of predicting the next token on internet text, which the paper argues is misaligned with the objective of helpfully and safely following user instructions.
A model trained only to predict the next word in internet text has no notion of what a user actually wants; it just learns what tends to come next. InstructGPT treats that mismatch as a design flaw worth fixing, not a minor side effect of scale.
The objective is explained on its own terms first, then the paper's complaint against it, before the thread runs to instruction following, the alternative the paper proposes.
The objective, in general
The language-modeling, or next-token-prediction, objective trains a model to assign high probability to the actual next token given everything before it. For a sequence $x_1,\ldots,x_T$ and model parameters $\theta$, training minimizes the negative log-likelihood, equivalently the cross-entropy between the model's predicted distribution and the observed next token, summed over the sequence: $$L(\theta) = -\sum_{t=1}^{T} \log P_\theta(x_t \mid x_{<t}).$$ This objective long predates InstructGPT, tracing through statistical n-gram language models, neural language models (Bengio et al., 2003), and recurrent and Transformer architectures, to the pretraining recipe used for GPT-2, GPT-3, and their contemporaries. InstructGPT inherits the objective as the standard way large language models, including GPT-3 (pretrained language model), are pretrained; it is not something the paper modifies but something it argues is insufficient on its own.
The paper's central complaint
InstructGPT's opening argument is that this objective, predicting the next token on internet text, is a different target from what users actually want: "making language models bigger does not inherently make them better at following a user's intent" (instructgpt, Abstract, p. 1), and outright, "the language modeling objective is misaligned" (instructgpt, §"1 Introduction", p. 2). Pursued alone, next-token prediction rewards plausible continuation of internet text, which need not be truthful, non-toxic, or helpful. The Language modeling (next-token prediction) objective — is argued to be misaligned with → Instruction following (as a fine-tuning objective) edge frames Instruction following (as a fine-tuning objective) as the paper's proposed corrective: a training signal built directly from human preference comparisons via Reinforcement learning from human feedback (RLHF), changing the objective itself rather than merely scaling the model trained under the old one. This concept anchors the Instruction following as an alternative to next-token prediction theme, which also situates two prior instruction-tuning baselines, FLAN (instruction-tuned baseline) and T0 (instruction-tuned baseline), as intermediate attempts to bridge the same gap by compilation rather than by preference learning.