Automatic overlap scores versus human and preference judgments — instruments only the extractive half of → The public NLP benchmark suite used to catch regressions

Only three of language-understanding-benchmarks' eight tasks are actually scored by an instrument this theme names. DROP, QuAC, and SQuAD v2 are free-response span-extraction tasks, each scored with quality-measurement-methods' F1 score against a target completion. HellaSwag, SST, RTE, and WSC are instead framed as multiple-choice completions and decided by a mechanism neither theme catalogues: "multiple choice answers are picked according to which one has the lowest average per-token log probability" (instructgpt, §"D Automatic evaluation details", p. 44), a comparison of the model's own output probabilities against itself, with no reference text and no overlap score computed. The overlap-metric family this theme supplies therefore instruments only the free-response half of the public benchmark suite; the multiple-choice half is settled by a probability comparison closer in kind to the entropy-based bias metric than to anything quality-measurement-methods defines.