Per-token KL penalty from the SFT model — densifies the terminal reward of → Bandit environment formulation for RLHF

In the bandit formulation the reward model contributes exactly one number per episode, delivered when the response is complete. The KL penalty is the objective's only per-token component: it is subtracted at every generated token, proportional to the log-ratio between the policy and the SFT model at that position (instructgpt, §"3.5 Models", p. 9). So beyond its stated purpose — mitigating over-optimization of the reward model — the penalty changes the shape of the learning problem: an otherwise sparse terminal-reward setting gains a dense signal that assigns credit at the same granularity as the actions the policy actually takes. Every token receives immediate feedback about whether it is drifting from the anchor distribution, while feedback about whether the answer is good arrives only at the end. The final models set β = 0.02 (instructgpt, §"C.4 Details of RLHF training", p. 42), keeping this dense term small — enough to shape token-level behavior without dominating the terminal reward that carries the actual preference information.