When a bug is traced to the labels, the repair goes to the labels too
A KL penalty can only pull a policy back toward its starting point; it cannot fix a reward model that was faithfully trained on the wrong preferences in the first place. Several separate failures across this corpus turn out to need that same lesson: the repair belongs in the data, or the labeling instructions behind it, not in the optimizer.
One paper solved an evasiveness problem by rewriting what labelers were told to reward, not by touching its RL step, and a hedging problem in another paper is the mirror case, a reward model doing exactly what its labels asked. An earlier paper shows the same logic on an advantage rather than a bug, a hard-coded labeling instruction doing work no reward function was asked to do, and the corpus's newest paper shows it once more on data selection itself, validating a screening method precisely on the messiest, least-curated real-world data available.
Five edges show a training pipeline's actual fix, or its actual source of an advantage, landing upstream, in the data or the labeling instructions, even when a plausible downstream, optimizer-level or reward-function-level explanation is available. The KL penalty was not the fix applied to evasiveness: it constrains how far PPO drifts from the SFT model but never touches the labeling instructions that rewarded refusal, so Constitutional AI's real fix was to change what crowdworkers were asked to prefer. Reward overoptimization surfaces as excessive hedging by the same logic, the KL penalty proving useless because the reward model was wrong "at home," faithfully optimizing labels that genuinely preferred hedged text, which is why the paper's proposed remedy is adversarial data collection rather than more regularization. And excessive hedging shares a fix but not a cause with false-premise failure, one an error of commission by the training signal, the other an error of omission by the data, yet both correctable only by touching the data, never the PPO machinery itself. The 2017 paper supplies a fourth instance five years earlier, an advantage rather than a bug traced to the same place: the Ant task's uprightness-first shaping is not an implicit preference raters independently discovered, but an upright priority hard-coded directly into contractor preference labeling's own instructions, ranking posture above forward speed, a rule absent from the task's true reward function, which treats uprightness as only a minor additive bonus. Where the other three edges show a downstream fix rerouted upstream into the data, this one shows the upstream data doing work no downstream, reward-function-level mechanism was ever asked to do. A fifth instance runs the same logic on data quality rather than a labeling instruction. Real-world chat data validation validates against LMSYS-Chat-1M precisely because that corpus is not carefully filtered or curated and contains a large fraction of toxic content, a deliberately messy, close-to-worst-case dataset rather than a clean benchmark; ranking its conversations by projection difference per trait and finetuning separately on the top-500, bottom-500, and 500 random samples produces a consistent ordering, high-projection samples inducing the strongest trait expression and low-projection samples the weakest, a result that survives even after GPT-4.1-mini strips out samples that overtly display the trait. Where the other four edges trace a bug or an advantage back to the labeling instructions themselves, this one shows the same upstream logic applied to data selection: the fix, or here the validation, lives in which real-world conversations get chosen for finetuning, not in any downstream training mechanism, and the effect is sharpest on LMSYS-Chat-1M precisely because it is the least curated of the four real-world datasets the appendix compares, the method's value clearest on data nobody has already cleaned.