Supervised fine-tuned (SFT) model
Training language models to follow instructions with human feedback — refined
A GPT-3 model fine-tuned on labeler demonstrations via supervised learning, used both as a standalone baseline and as the initialization for reward-model and PPO training.
Take GPT-3, show it thousands of human-written demonstrations of good responses, and fine-tune on them directly: the result is the SFT model, the first trained artifact in the InstructGPT (2022) pipeline. It serves both as a baseline in its own right and as the seed for everything trained afterward.
From there the page moves through how the model is trained, an overfitting curve explained by the dataset's construction, its selection by the reward model it later initializes, and the way it quietly forks in two.
What it is and how it's trained
The SFT model is a GPT-3 model fine-tuned directly on labeler demonstrations using ordinary supervised learning, for 16 epochs with cosine learning-rate decay and a residual dropout of 0.2 (instructgpt, §"3.5 Models", p. 8). It plays a dual role in the corpus: a standalone baseline evaluated in its own right, and the initialization from which both the Reward model (RM) and the PPO fine-tuning stage (RLHF policy optimization) policy are built.
An overfitting curve explained by how the dataset was built
The SFT dataset — duplication underlies early overfitting of → Supervised fine-tuned (SFT) model edge connects a puzzling training result to a fact about data construction: the SFT model's validation loss overfits after just one epoch, yet the paper trains for sixteen anyway because later epochs keep improving both RM score and human preference (instructgpt, §"C.1 Details of SFT training", p. 40). A likely contributor is that the SFT dataset's roughly 13k examples are not 13k independent demonstrations; because instruction-like prompts were scarce early on, labelers wrote template instructions with several few-shot examples each, and multiple training datapoints were then synthetically constructed by resampling different subsets of the same few-shot set, yielding 11,295 labeler-written examples against only 1,430 customer-derived ones (instructgpt, §"A.3 Dataset sizes", p. 33).
Selected by the reward model it later initializes
Because more epochs helped RM score and human preference despite hurting validation loss, the final SFT checkpoint was chosen by its score under the reward model rather than by validation loss, the paper stating plainly that RM score "proved more predictive of human preference results than validation loss" (instructgpt, §"C.1 Details of SFT training", p. 41), a relationship the Reward model (RM) — back selects the checkpoint of → Supervised fine-tuned (SFT) model edge names. This means the "supervised baseline" that PPO is later measured against was itself already selected to please the preference model, understating how much the reward model shapes the entire pipeline, and giving the reward model a role in model selection before it ever produces a single RL reward.
A model that quietly forks in two
The Supervised fine-tuned (SFT) model — anchors as both init and reference for → PPO fine-tuning stage (RLHF policy optimization) edge reveals that "the SFT model" is not a single artifact once PPO enters the picture: the checkpoint reported as the SFT baseline was trained for 16 epochs, while the checkpoint used to initialize the RL policy, and to serve as the reference distribution for the Per-token KL penalty from the SFT model, was a separate variant trained for only 2 epochs with 10% pretraining data mixed in, chosen because sweeps showed this was the only initialization setting that stood out for PPO performance (instructgpt, §"C.3 Details of the initialization models for RLHF", p. 42; §"E.8 PPO init models", p. 56). This concept sits at the intersection of the InstructGPT's three-step training pipeline theme and the Learning from demonstrations instead of a specified reward theme, which traces Supervised fine-tuning (SFT) back to Concrete Problems' proposal to learn from demonstrations rather than a specified reward.