Language modeling (next-token prediction) objective — is argued to be misaligned with → Instruction following (as a fine-tuning objective)

InstructGPT's opening argument is that the standard pretraining objective — predicting the next token on internet text — and the objective users actually want are different targets, and that scaling up the former does not close the gap: 'making language models bigger does not inherently make them better at following a user's intent' (instructgpt, Abstract, p. 1). Pursuing the language-modeling objective alone produces outputs that are untruthful, toxic, or simply unhelpful, because next-token prediction rewards plausible continuation of internet text rather than service to the user; the paper states outright that 'the language modeling objective is misaligned' (instructgpt, §"1 Introduction", p. 2). Instruction following is defined and trained as the corrective: a fine-tuning objective built directly from human preference comparisons on real user-submitted prompts via RLHF, so the training signal itself changes rather than merely the model's scale.