SFT dataset
Training language models to follow instructions with human feedback — introduced
The set of about 13k training prompts (labeler-written and API-sourced) paired with human demonstrations, used to train the supervised fine-tuned models.
Roughly thirteen thousand prompts, each paired with an answer a person actually wrote: that is the raw material for the first stage of InstructGPT's training. Its contents came partly from hired labelers and partly from real users of an earlier model.
It traces back first to a proposal sketched in Concrete Problems in AI Safety (2016), then forward to what it enabled downstream, including a data quirk worth knowing before trusting the numbers built on top of it.
How it fits — the concrete descendant of a 2016 proposal
The SFT dataset is where Concrete Problems' abstract 2016 proposal to learn from demonstrations instead of a hand-specified reward becomes, six years later, the concrete first stage of a real alignment pipeline. Concrete Problems considers inverse reinforcement learning, apprenticeship learning, and cooperative inverse reinforcement learning as ways an agent might learn a policy from expert trajectories rather than trial-and-error optimization of a specified objective. InstructGPT operationalizes exactly this idea, in its structurally simplest form, as Supervised fine-tuning (SFT): training a pretrained GPT-3 model directly on roughly 13,000 human-written demonstrations of desired behavior to produce the Supervised fine-tuned (SFT) model, used both as a standalone baseline and as the initialization for everything downstream (Learning from demonstrations instead of a specified reward theme; instructgpt, §"3.2 Dataset", p. 7). Notably, the safety property the 2016 paper attached to demonstration learning — bounding further exploration around the resulting baseline policy — does not travel with it into SFT itself; supervised fine-tuning has no exploration to bound, and that bound reappears two pipeline stages later as the PPO-stage Per-token KL penalty from the SFT model instead (Learning from demonstrations instead of a specified reward — displaces its exploration bound onto → The mechanics that make PPO fine-tuning work).
What it enabled downstream, and a data quirk worth knowing
The SFT dataset does more than train one model: it trains the policies whose outputs get sampled to build the RM dataset, since most comparison data for reward-model training comes from these supervised policies (instructgpt, §"3.1 High-level methodology", p. 6). This means the reward model's competence is local to the neighborhood of outputs SFT-style policies actually produce — a fact the later KL penalty depends on, since keeping PPO's policy near the SFT model also keeps it inside the region where the reward model's training data lives. The dataset also bootstrapped the API prompt distribution InstructGPT is now evaluated on: users learned to submit instruction-style prompts by interacting with an earlier InstructGPT model that had already been supervised-fine-tuned on labeler demonstrations, making the "real usage" distribution a co-evolved artifact rather than independent ground truth (instructgpt, §"3.2 Dataset", p. 6-7). One further wrinkle: roughly two-thirds of the dataset's apparent 13k examples are synthetic resamples of the same labeler-written templates rather than independent demonstrations, since customer prompts resembling instructions were scarce early on and labelers instead wrote template instructions with several few-shot examples that were then split into multiple training points — a plausible contributor to the SFT model overfitting validation loss after just one epoch, and the reason final SFT checkpoints are selected by RM score rather than validation loss (instructgpt, §"A.3 Dataset sizes", p. 33; §"C.1 Details of SFT training", p. 40).