F1 score (span-overlap QA metric) — diverges under rlhf from → Win rate (preference against baseline)

These two metrics deliver opposite verdicts on the same intervention, and InstructGPT's methodology is largely a negotiation between them. PPO fine-tuning pushes win rate to its headline heights — 85% preference over GPT-3 — while collapsing span-overlap F1 on extractive QA: zero-shot SQuADv2 falls from 64.30 for 175B GPT-3 to 43.68 for PPO, and DROP from 27.53 to 13.08 (instructgpt, §"E.1 Performance on public NLP datasets", Table 14, p. 56). The paper resolves the disagreement asymmetrically: win rate is treated as the objective and F1 as a constraint. The pretraining-mix coefficient that defines PPO-ptx was tuned specifically to recover SQuADv2 and DROP F1 while leaving validation reward and Likert scores intact (instructgpt, §"E.6 Fixing regressions on public NLP datasets", p. 53). In effect, F1 is demoted within a single paper from a quality measure to a regression alarm: it no longer adjudicates which model is better — human preference does — but it still detects when alignment quietly spends capability that preference evaluation cannot see.