Alignment tax
Training language models to follow instructions with human feedback — introduced
The performance cost, in terms of regressions on public NLP benchmarks, incurred as a side effect of an alignment fine-tuning procedure such as RLHF.
Making a model more aligned can quietly make it less capable. InstructGPT found that its fine-tuned models slipped on standard NLP benchmarks even as human raters preferred their answers, a cost the paper names the alignment tax.
The page places the idea among the paper's own admitted shortfalls, explains why noticing it matters, and finally gets concrete about what it looks like in practice. It connects most directly to reward-model over-optimization, the pathology it sits beside.
How it fits
Alignment tax names the second of InstructGPT's two documented RLHF pathologies, sitting alongside Reward model over-optimization in the paper's own account of where its alignment "still falls short" even as its headline preference metrics improve. Where over-optimization is measured in preference, the tax is measured in a different currency entirely: regressions on standard public NLP benchmarks such as DROP and SQuAD v2 that RLHF fine-tuning introduces as a side effect of steering the model toward instruction-following (instructgpt, §"4.2 Results on public NLP datasets", p. 14; §"5.1 Implications for alignment research", p. 17). InstructGPT introduces both the term and the phenomenon as it applies to RLHF.
What it enables and why it matters
The concept motivates one of the paper's specific engineering fixes: PPO-ptx, which mixes pretraining language-modeling gradients into the PPO update. The paper frames a low alignment tax as important on strategic grounds, not just technical ones — if alignment techniques degrade general capability too much, aligned models will be less competitive and less likely to be adopted, undermining the whole point of building them. The tax and reward-model over-optimization demand different cures even though both stem from the same act of pushing PPO hard against a learned objective: raising the KL-penalty coefficient $\beta$ to fight the tax costs validation reward without ever fully recovering DROP or SQuADv2, whereas tuning the pretraining-mix coefficient $\gamma$ reverses the regressions at minimal cost (instructgpt, §"4.2 Results on public NLP datasets", p. 15). Both pathologies worsen with more optimization pressure — doubling RL training from 256k to 512k episodes deepens the benchmark regressions (instructgpt, §"E.6 Fixing regressions on public NLP datasets", p. 56).
What it is, concretely
Measured across the public NLP benchmark suite and the translation and summarization benchmarks, the tax is not uniform: it hits free-generation and span-extraction tasks harder than ranking-style multiple-choice tasks, and PPO-ptx does not fully erase it everywhere, leaving DROP, SQuADv2, and WMT translation as the paper's short list of unresolved regressions even after mitigation (instructgpt, §"4.2 Results on public NLP datasets", p. 15). The name has stuck in later alignment discussion as shorthand for any capability cost paid for an alignment intervention, extending well beyond RLHF or this specific paper.