When a benchmark stops measuring and starts being optimized against
A benchmark that starts out just checking a model can end up as the very thing a training run gets tuned against, and once that happens, saying the alignment tax was paid down means something narrower than it sounds. InstructGPT's own numbers show that shift happening more than once.
The page tracks SQuAD and DROP functioning as one shared checkpoint for two different tuning sweeps, notes that a multiple-choice task recovers where a free-generation one does not, and points out translation absorbing a cost that summarization escapes entirely. It ends on F1's quiet reassignment: no longer a scorecard for quality, just a tripwire for regression.
Several of InstructGPT's benchmarks are quietly promoted from neutral evaluation to tuning target, and the promotion changes what "the alignment tax was fixed" actually means. SQuADv2 forms the regression testbed with DROP: both the pretraining-coefficient sweep and the KL-coefficient sweep are judged specifically on this pair. Hellaswag escapes the regression that persists in DROP because it is a ranking task rather than a free-generation task, showing the recoverable part of the tax sits in likelihood rankings while the persistent part lives in what the model chooses to write. WMT15 fr-en bears the alignment tax that spares Reddit TL;DR, a split that tracks distance from the RLHF fine-tuning distribution rather than task difficulty. And F1 score diverges under RLHF from win rate so sharply that the paper demotes F1 from a quality measure to a regression alarm, no longer adjudicating which model is better but still catching what preference evaluation cannot see.