SQuAD v2
Training language models to follow instructions with human feedback — inherited
A reading-comprehension QA benchmark (Rajpurkar et al., 2018) that includes unanswerable questions, used as one of the public NLP datasets showing PPO regressions.
The trick in SQuAD v2 is that some of its questions have no answer in the passage at all. A model must not only find answer spans but recognize when there is nothing to find, a small honest skill layered onto ordinary reading comprehension.
Set beside its sibling benchmarks in the same regression-catching suite, what it revealed under PPO fine-tuning is the more interesting half, with DROP the closest point of comparison.
How it fits
SQuAD v2 (Rajpurkar et al., 2018) extends the original Stanford Question Answering Dataset with unanswerable questions, requiring a model to recognize when no answer exists in the given passage rather than always producing a span. It is inherited into InstructGPT's public NLP benchmark suite as one of eight academic comprehension tasks used to detect the Alignment tax, and it predates this paper as a standard reading-comprehension benchmark (instructgpt, §"1 Introduction", p. 3; §"SquadV2 (Stanford Question Answering Dataset)", p. 47).
What it revealed
SQuAD v2's central role in the corpus is as one half of a promoted tuning target: it forms the regression testbed with DROP, since InstructGPT's authors used exactly this pair — "the datasets we used for testing" — to select both the pretraining-loss coefficient $\gamma$ and the KL-penalty coefficient $\beta$ in Equation 2 (instructgpt, §"4.2 Results on public NLP datasets", p. 15). Raising the KL coefficient a hundredfold never fully recovers either benchmark's regression, while a pretraining coefficient of 27.8 does, at minimal cost elsewhere (instructgpt, §"E.6 Fixing regressions on public NLP datasets", p. 53). The consequence is a scoping caveat worth stating plainly: when the paper reports that PPO-ptx "mostly mitigates" the alignment tax, that claim operationally means F1 was restored specifically on this pair of extractive QA sets — exact-span answers scored by token overlap via F1 score (span-overlap QA metric) — not that general capability loss was measured and repaired across the board. SQuADv2 stays on the paper's list of benchmarks PPO-ptx does not fully recover, alongside DROP and WMT translation (instructgpt, §"4.2 Results on public NLP datasets", p. 15).