RealToxicityPrompts
Training language models to follow instructions with human feedback — inherited
A benchmark dataset (Gehman et al., 2020) of sentence-start prompts with varying toxicity, used to evaluate a model's propensity to continue text toxically.
Give a model the start of a sentence and see whether it finishes politely. RealToxicityPrompts supplies sentence openings at varying levels of toxicity, so a model's tendency to continue text toxically can be measured rather than guessed at.
The page situates the benchmark within InstructGPT's safety evaluations, then looks at how its ground truth gets defined and what that costs. Reading it alongside Winogender, its connected neighbor, shows what the two revealed together that neither could alone.
How it fits
RealToxicityPrompts (Gehman et al., 2020) is a benchmark of sentence-start prompts spanning a range of toxicity levels, inherited into InstructGPT's Operationalizing truthfulness and social bias into measurable benchmarks to test whether a model continues text toxically. It predates this paper as an established toxicity-evaluation dataset (instructgpt, §"3.6 Evaluation", p. 10; §"RealToxicityPrompts", p. 45).
How its ground truth is defined, and what that costs
The Perspective API supplies this benchmark's ground truth: whether a continuation counts as toxic is decided by a proprietary, third-party classifier that is retrained over time, which the paper calls "the standard evaluation procedure for this dataset" (instructgpt, §"4.2 Results on public NLP datasets", p. 13). InstructGPT is unusually careful about auditing this dependency — samples for 1,729 prompts were also rated by human labelers for absolute toxicity, toxicity relative to the prompt, continuity, and overall preference, and the human and automatic evaluations agreed on the qualitative pattern (instructgpt, §"4.2 Results on public NLP datasets", p. 14). A separate methodological caveat compounds the dependency on Perspective's classifier: prompts were sampled roughly uniformly by prompt toxicity rather than by the dataset's natural distribution, which inflates the paper's absolute toxicity numbers relative to standard use and makes them incomparable to other published RealToxicityPrompts results (instructgpt, §"4.2 Results on public NLP datasets", p. 14).
What it revealed alongside a different safety benchmark
RealToxicityPrompts is linked by an unusually direct experimental design to Winogender: it diverges under respectful prompting from Winogender, since it reuses Winogender's exact instruction protocol — the same basic, "respectful," and "biased" prompt variants (instructgpt, §"RealToxicityPrompts", p. 45). Under the respectful instruction, toxicity falls: InstructGPT generates measurably less toxic continuations than GPT-3, an advantage that vanishes without the instruction. On Winogender the same instruction lowers entropy under the Entropy-based bias metric, which that metric reports as stronger bias. One safety benchmark improves under precisely the intervention that makes the other worse, which is the corpus's clearest demonstration that its safety benchmarks do not form a single axis.