Per-token KL penalty from the SFT model — was not the fix applied to → Evasiveness

Both concepts are engineered countermeasures to a Goodhart-style failure in RLHF, but they intervene at opposite ends of the training pipeline, and comparing them shows that Goodharting in RLHF has at least two independent entry points. The KL penalty acts downstream, inside the optimization step: it does not touch the reward model or the human labels that produced it, it simply constrains how far the PPO policy is allowed to drift from the SFT model while chasing RM score (instructgpt, §"3.5 Models", p. 9). Evasiveness was not fixed that way. Its cause was traced upstream, to the labeling instructions themselves: earlier crowdworkers were told only to pick the more harmless of two responses, which "likely produced a significant amount of data favoring evasiveness," so Constitutional AI's fix was to change what workers were asked to prefer, choosing the more nuanced, transparent response over the evasive one whenever both were similarly harmless (constitutional-ai, §"4.4 Harmlessness vs. Evasiveness", p. 13). One bug was patched by constraining the optimizer; the other, by correcting the training signal before optimization ever saw it.