F1 score (span-overlap QA metric)
Training language models to follow instructions with human feedback — inherited
A token-overlap-based scoring metric used to evaluate model-generated answers against target completions on DROP, QuAC, and SQuAD v2.
If an answer is a span of text, its quality can be scored by token overlap: how many words the model's answer shares with the target, balanced between precision and recall. That balance is the F1 score, applied here to three question-answering benchmarks.
The page relates the metric to the corpus's broader family of quality measurements, then reviews what it revealed across DROP, QuAC, and SQuAD v2. Win rate, the human-judgment metric it contrasts with, makes the natural counterpoint to read next.
How it fits
F1 score is the automatic overlap metric InstructGPT uses to score free-response span-extraction tasks — DROP, QuAC, and SQuAD v2 — inside the broader Automatic overlap scores versus human and preference judgments family that spans both InstructGPT and Constitutional AI, alongside ROUGE-L for summarization and BLEU for translation. It is inherited rather than invented here: F1, the harmonic mean of precision and recall over predicted-versus-target token overlap, is a standard information-retrieval and QA metric that long predates this paper (instructgpt, §"DROP (Discrete Reasoning Over the Content of Paragraphs)", p. 46).
What it revealed and what it enabled
Inside InstructGPT, F1's most consequential role is not scoring per se but disagreeing sharply with the paper's primary metric: F1 diverges under RLHF from win rate. PPO fine-tuning pushes win rate to its headline heights, 85% preference over GPT-3, while simultaneously collapsing F1 on extractive QA — zero-shot SQuADv2 falls from 64.30 to 43.68, and DROP from 27.53 to 13.08 (instructgpt, §"E.1 Performance on public NLP datasets", Table 14, p. 56). The paper resolves this disagreement asymmetrically rather than symmetrically: win rate is treated as the actual training objective and F1 as a constraint to be satisfied afterward, and the pretraining-mix coefficient defining PPO-ptx was tuned specifically to recover SQuADv2 and DROP F1 while leaving validation reward and Likert scores untouched (instructgpt, §"E.6 Fixing regressions on public NLP datasets", p. 53). Within a single paper, F1 is thereby demoted from a metric that adjudicates model quality to a regression alarm: it no longer decides which model is better, but it still catches capability loss that human preference evaluation, by construction, cannot see.