Supervised fine-tuned (SFT) model — anchors as both init and reference for → PPO fine-tuning stage (RLHF policy optimization)

The SFT model plays two distinct roles in the PPO stage: the policy starts from its weights, and the per-token KL term in the training objective is computed against it, so the same artifact defines both where optimization begins and what it is pulled back toward (instructgpt, §"3.5 Models", p. 9). Less visibly, "the SFT model" here is not the checkpoint evaluated as the SFT baseline. The RL initialization was a variant trained for 2 epochs on demonstrations with 10% pretraining data mixed in, while the flagship SFT baseline was trained for 16 epochs; the 10%-mix setting was chosen because sweeps showed it was the only initialization variant that stood out for PPO performance (instructgpt, §"C.3 Details of the initialization models for RLHF", p. 42; §"E.8 PPO init models", p. 56). So the pipeline's second stage quietly forks: one SFT lineage exists to be compared against, another exists to be optimized from, and the anchor for the KL penalty is the latter.