DROP (Discrete Reasoning Over the content of Paragraphs)
Training language models to follow instructions with human feedback — inherited
A reading-comprehension benchmark (Dua et al., 2019) requiring discrete reasoning (e.g., counting, sorting) over paragraphs, used as one of the public NLP datasets showing PPO regressions.
Reading comprehension is one thing; counting, sorting, and doing small arithmetic over what you read is another. DROP demands that second, more discrete kind of reasoning about paragraphs, and InstructGPT uses it to watch for capability slips.
It sits inside the eight-task public benchmark suite built to catch regressions, but what it actually revealed about the fine-tuned models is the part worth reading closely, alongside its neighbors HellaSwag and SQuAD v2.
How it fits
DROP, Discrete Reasoning Over the content of Paragraphs (Dua et al., 2019), is one of eight tasks in InstructGPT's public NLP benchmark suite, a battery of academic comprehension benchmarks the paper did not design but runs its models against specifically to catch the Alignment tax. It requires discrete reasoning over a paragraph — counting, sorting, addition — rather than simple span lookup, and predates InstructGPT as an established reading-comprehension diagnostic (instructgpt, §"1 Introduction", p. 3; §"DROP (Discrete Reasoning Over the Content of Paragraphs)", p. 46).
What it revealed
DROP and SQuAD v2 are effectively promoted from passive evaluation to active tuning target: the two form a single regression testbed, since both the pretraining-coefficient sweep and the KL-coefficient sweep were judged specifically on "SQuADv2 and DROP (the datasets we used for testing)" (instructgpt, §"4.2 Results on public NLP datasets", p. 15). A pretraining loss coefficient of 20 or more recovers their regressions at 1.3B scale, and the adopted production value of 27.8 was validated against exactly this pair (instructgpt, §"E.6 Fixing regressions on public NLP datasets", p. 53). DROP is also one side of a revealing split: HellaSwag escapes the regression that persists in DROP, because HellaSwag is scored by ranking candidate endings by log probability while DROP requires sampling free-form text and scoring token overlap via F1 score (span-overlap QA metric) — the recoverable part of the alignment tax sits in the model's likelihood rankings, while the part that persists after PPO-ptx lives in what the model actually chooses to write (instructgpt, §"4.2 Results on public NLP datasets", p. 15). DROP remains on the paper's short list of unfixed regressions even after mitigation, alongside SQuADv2 and WMT translation.