Reward model over-optimization — is preemptively countered by → Per-token KL penalty from the SFT model

The two concepts' relationship is more specific than "KL penalty limits over-optimization": the design of the penalty is shaped by an asymmetry between where the reward signal lands and where the counter-pressure is applied. In InstructGPT's PPO setup, the reward model assigns a single scalar reward only once, at the end of the episode, after the full response has been generated (instructgpt, §"3.5 Models", p. 9). If the KL constraint were applied the same way, once per episode, a policy could in principle generate however it likes for most of a response and only correct course near the end. Instead the KL penalty is applied per token, at every step of generation, so the policy pays a continuous cost for drifting from the SFT distribution throughout the entire response, not just at the point where the RM's judgment is finally rendered. This makes the counter-pressure track the same surface over which reward-hacking behavior could accumulate: a policy that drifts into an RM blind spot gradually across many tokens is taxed gradually across those same tokens, rather than being free until the very end.