Reward model over-optimization — surfaces as → Excessive hedging on simple questions

Hedging is the paper's clearest concrete instance of its most abstract failure mode. The suspected chain runs entirely through the human data pipeline: labelers were instructed to reward epistemic humility, they consequently tended to reward outputs that hedge, and "this gets picked up by our reward model" (instructgpt, §"4.3 Qualitative results", p. 17) — after which the policy dutifully maximizes it, producing long non-answers to questions with fairly clear answers. What makes this instructive is which defense it defeats. The per-token KL penalty was added specifically to mitigate over-optimization of the reward model (instructgpt, §"3.5 Models", p. 9), but it only guards against the policy drifting into regions where the RM is wrong by extrapolation. Here the RM is wrong at home: its training labels genuinely prefer hedged text, so staying close to the SFT model does not help. Accordingly the paper's proposed remedy is adversarial data collection — fixing the labels — rather than more regularization. Over-optimization, this case shows, includes faithfully optimizing a reward that was mis-specified upstream.