Per-token KL penalty from the SFT model
Training language models to follow instructions with human feedback — inherited
A term subtracted from the reward at every generated token, proportional to the KL divergence between the RL policy and the SFT model, used to keep the policy close to the SFT model and limit reward over-optimization.
A model being optimized against a learned reward will happily wander into strange territory to please it. The per-token KL penalty is the leash: at every token, the policy pays a small cost for straying from the supervised model it started as.
What follows moves from what a KL penalty is in general to why this one bites at every token rather than once per episode, then out to the corpus's wider story about proxies going wrong. Its tradeoff against pretraining-gradient mixing is the part worth lingering on.
What a KL penalty is
The Kullback-Leibler divergence between two distributions $P$ and $Q$ over the same space is $D_{KL}(P \| Q) = \sum_x P(x)\log\frac{P(x)}{Q(x)}$, a non-negative, asymmetric measure of how much $P$ diverges from $Q$. Constraining or penalizing this divergence between a new policy and a reference policy during optimization predates this paper by years, underlying trust-region methods such as TRPO and forming a standard regularization term in PPO-style RL fine-tuning of language models (as in Ziegler et al., 2019, and Stiennon et al., 2020, which InstructGPT explicitly follows). The concept is inherited: InstructGPT does not invent the KL-penalty idea but applies it in a specific, per-token form.
Applied per token, not per episode
InstructGPT subtracts, from the reward at every generated token, a term $-\beta \log\big(\pi^{RL}_\phi(y|x)/\pi^{SFT}(y|x)\big)$ proportional to the log-ratio between the RL policy and the Supervised fine-tuned (SFT) model at that position (instructgpt, §"3.5 Models", p. 9; Equation 2), with the final models setting $\beta = 0.02$ (instructgpt, §"C.4 Details of RLHF training", p. 42). As the Per-token KL penalty from the SFT model — densifies the terminal reward of → Bandit environment formulation for RLHF edge observes, this design choice does more than its stated job: because the Reward model (RM) delivers exactly one scalar per episode in the Bandit environment formulation for RLHF, only at the very end, the per-token KL term is the objective's sole source of dense, mid-generation feedback, kept deliberately small so it shapes token-level behavior without swamping the terminal reward that carries the actual preference signal.
Its place in the corpus's Goodhart story
The Reward model over-optimization — is preemptively countered by → Per-token KL penalty from the SFT model edge frames the penalty's core purpose precisely: it does not touch the reward model or the human labels behind it, only how far the PPO policy may drift from the SFT model while chasing RM score, applying continuous pressure across the same tokens over which reward-hacking behavior could accumulate. But the penalty is not a universal fix for Goodhart's law in this corpus. The Per-token KL penalty from the SFT model — was not the fix applied to → Evasiveness edge contrasts it with Constitutional AI's Evasiveness problem, whose cause was traced upstream to the labeling instructions themselves rather than downstream to the optimizer, so its fix changed what crowdworkers were asked to prefer rather than constraining the RL policy. Excessive hedging is the case where the KL penalty demonstrably fails within InstructGPT itself: because the reward model's training labels genuinely prefer hedged text, staying close to the SFT distribution offers no protection, and the RM is simply wrong at home.
A tradeoff against pretraining-gradient mixing
The PPO-ptx (pretraining gradient mixing) — outperforms tightening of → Per-token KL penalty from the SFT model edge reports a direct experimental comparison: raising $\beta$ to counter public-NLP-dataset regressions sacrifices substantial validation reward and never fully recovers performance on DROP and SQuADv2, whereas a well-chosen PPO-ptx (pretraining gradient mixing) pretraining-mix coefficient reverses those regressions at minimal reward cost (instructgpt, §"4.2 Results on public NLP datasets", p. 15). The asymmetry reflects what each term actually does: the KL penalty constrains where the policy may go relative to the SFT model, so buying back capability with it costs alignment gain proportionally, whereas pretraining gradients inject the missing training signal directly. Pretraining-gradient mixing does not make the KL knob redundant, however; even with ptx active, human Likert scores peak near $\beta \approx 0.01$–$0.02$ and degrade at both $\beta=0$ and $\beta=2$ (instructgpt, §"E.7 Optimal KL reward coefficient", p. 56). The concept belongs to both the The mechanics that make PPO fine-tuning work theme and the cross-corpus Goodhart's law across the corpus: when the proxy becomes the target theme.