CNN/Daily Mail summarization
Training language models to follow instructions with human feedback — inherited
A news-summarization benchmark (Nallapati et al., 2016) using a 'TL;DR:' prompting format, included in the automatic evaluation suite.
Summarizing news articles is a classic test of generation, and this benchmark, CNN and Daily Mail stories prompted with a plain TL;DR tag, is InstructGPT's news-summarization entry, scored by overlap with reference summaries.
The page situates it among the generation benchmarks before turning to something less routine: what evaluation here uncovered was a data-integrity problem, not just a middling score. Its Reddit counterpart shares the same issue in a different dataset.
How it fits
CNN/Daily Mail summarization (Nallapati et al., 2016) is a news-summarization benchmark, prompted with a "TL;DR:" format and scored via ROUGE-L against reference summaries, inherited into InstructGPT's automatic evaluation suite within Testing generative capability: translation and summarization benchmarks alongside Reddit TL;DR summarization (instructgpt, §"D Automatic evaluation details", p. 44; §"CNN/DM Summarization", p. 49).
What it revealed: a data-integrity problem, not just a measurement
This benchmark's most notable role in the corpus is as one half of an apparent reporting error: it duplicates the reported results of Reddit TL;DR summarization. The two benchmarks were meant to be independent checks — professionally-written news summaries versus author-written Reddit summaries — but in Table 14 the CNN/DM and TL;DR rows are numerically identical across all twelve models, to three decimal places (0.182 through 0.220), which is implausible for genuinely distinct datasets and almost certainly reflects a duplicated row rather than two independent measurements (instructgpt, §"E.1 Performance on public NLP datasets", p. 56). The copy-paste trail extends into Appendix D: both benchmarks' dataset-features boxes describe their few-shot setting as containing "15 additional French / English pairs," text apparently carried over from the WMT translation figure (instructgpt, §"CNN/DM Summarization", p. 49; §"TLDR Summarization", p. 50). The practical consequence for a reader is concrete: InstructGPT's ROUGE-L evidence on summarization is one measurement, not two independent confirmations, and anyone citing the paper's summarization scores should first establish which dataset the surviving numbers actually belong to, since at least one of the two labeled rows cannot be what it claims to be.