WMT 2015 French-to-English translation — bears the alignment tax that spares → Reddit TL;DR summarization

Both tasks demand fluent open-ended generation, yet RLHF taxes one and helps the other, and the split maps onto distance from the fine-tuning distribution rather than onto difficulty. PPO drops 175B zero-shot BLEU on WMT from 38.92 to 24.16, and even PPO-ptx recovers only to 34.28 — translation stays on the short list of unfixed regressions alongside DROP and SQuADv2 (instructgpt, §"4.2 Results on public NLP datasets", p. 15) — while TL;DR ROUGE-L rises above the GPT-3 baseline under the same training, 0.196 to 0.227 for PPO (Table 14, p. 56). The asymmetry is explicable: the API prompt data that drives RLHF is over 96% English (instructgpt, §"3.3 Tasks", p. 7), and summarization is a named API use-case whose TL;DR prompt format mirrors real usage — Table 1 lists it at 4.2% of prompts (p. 6) — indeed Reddit TL;DR is the very domain where the paper's RLHF method was developed by Stiennon et al. (2020). French-to-English translation is essentially unrepresented in that distribution. Generative capability transferred where the benchmark resembled the tuning data, and was spent where it did not.