Learning from demonstrations instead of a specified reward — displaces its exploration bound onto → The mechanics that make PPO fine-tuning work

The 2016 proposal pairs demonstration learning with an explicit safety idea absent from this theme's account of SFT: bounding further exploration around the baseline policy that demonstrations produce, so a CIRL- or apprenticeship-trained agent does not wander far from validated behavior. Supervised fine-tuning, the pipeline step this theme's narrative says operationalizes demonstration learning, carries no such bound; it is unconstrained supervised learning with nothing to bound, since there is no exploration at that stage. The bound reappears two pipeline steps later, in a different theme, as the per-token KL penalty subtracted from reward during PPO fine-tuning, which keeps the policy from drifting too far from the SFT model (instructgpt, §3.5 Models, p. 9). The safety property 2016 attached to demonstration learning itself is, in the built pipeline, detached from SFT and re-implemented downstream as a separate PPO-stage regularizer.