Perspective API
Training language models to follow instructions with human feedback — inherited
A third-party automatic toxicity-scoring service used to obtain toxicity scores for model outputs on RealToxicityPrompts, cross-checked against human toxicity ratings.
Toxicity scores in InstructGPT's evaluations come from an outside judge: Perspective API, a service from Google's Jigsaw that has rated the toxicity of text since 2017, trained on large volumes of online comments rated by crowdworkers.
Its limits matter as much as its scores, and the paper checks both against human toxicity ratings directly. RealToxicityPrompts, the benchmark it judges, is the natural place to see the API at work.
What the service is
Perspective API is a machine-learning-based toxicity-scoring service, developed by Google's Counter Abuse Technology team and Jigsaw and made publicly available beginning in 2017, that returns probability scores for attributes such as "toxicity" given a piece of text, trained on large volumes of online comments rated by crowdworkers. It predates InstructGPT by roughly five years and is inherited here as a third-party, off-the-shelf classifier rather than anything the paper builds itself.
Its role, and its limits, in this corpus
InstructGPT uses Perspective API to score model outputs on the RealToxicityPrompts benchmark, calling this "the standard evaluation procedure for this dataset" (instructgpt, §"4.2 Results on public NLP datasets", p. 13). The Perspective API — supplies the ground truth for → RealToxicityPrompts edge highlights what this dependency costs the paper's claims: since ground truth is outsourced to a proprietary, periodically retrained third-party classifier, its verdicts are neither inspectable nor stable across papers. InstructGPT is unusually careful about auditing this: 1,729 prompts were also rated by human labelers for absolute toxicity, toxicity relative to the prompt, continuity, and overall preference, and the human and automatic evaluations agreed on the qualitative pattern, that InstructGPT beats GPT-3 on toxicity only under a "respectful" instruction (instructgpt, §"4.2 Results on public NLP datasets", p. 14). A further caveat compounds the dependency: prompts were sampled roughly uniformly by toxicity rather than by the dataset's natural distribution, so the paper's absolute Perspective scores are inflated and not comparable to other published RealToxicityPrompts numbers (instructgpt, §"4.2 Results on public NLP datasets", p. 14). Perspective API is a member of the Operationalizing truthfulness and social bias into measurable benchmarks theme, the instrument that makes toxicity measurable at all in this corpus.