Perspective API — supplies the ground truth for → RealToxicityPrompts

RealToxicityPrompts ships prompts, not judgments: whether a continuation counts as toxic is decided by the Perspective API, and the paper calls running samples through it "the standard evaluation procedure for this dataset" (instructgpt, §"4.2 Results on public NLP datasets", p. 13). The benchmark's ground truth is thus outsourced to a proprietary third-party classifier that is retrained over time, so its verdicts are neither inspectable nor stable across papers. InstructGPT is unusual in auditing that dependency: samples for 1,729 prompts were also rated by human labelers — for absolute toxicity, toxicity relative to the prompt, continuity, and overall preference — and the human and automatic evaluations agreed on the qualitative pattern, namely that InstructGPT beats GPT-3 only when given a "respectful" instruction (instructgpt, §"4.2 Results on public NLP datasets", p. 14). A further caveat compounds the dependency: prompts were sampled approximately uniformly by prompt toxicity rather than by the dataset's natural distribution, so the paper's absolute Perspective scores are inflated and cannot be compared against other published RealToxicityPrompts numbers (p. 14).