SFT dataset — trains the policies sampled for → RM (comparison) dataset

The RM dataset's labels are rankings, but its inputs are model completions, and those completions come overwhelmingly from policies trained on the SFT dataset: "most of our comparison data comes from our supervised policies, with some coming from our PPO policies" (instructgpt, §"3.1 High-level methodology", p. 6). The demonstration data therefore controls not just what the initial policy does but what the reward model ever gets to see. The RM learns human preferences over the neighborhood of SFT-style outputs — its competence is local to that region of output space. This dovetails with the KL penalty's role in the RL stage: keeping the policy near the SFT model also keeps it inside the region where the RM's training data lives, so the constraint that limits over-optimization simultaneously keeps the judge on-distribution. The paper notes that steps 2 and 3 can be iterated, with fresh comparisons collected on the current best policy — the mechanism by which the RM's competent region can track a moving policy — though in this paper most comparison data still comes from the supervised policies.