Operationalizing truthfulness and social bias into measurable benchmarks — falls outside the taxonomy of → Automatic overlap scores versus human and preference judgments

Quality-measurement-methods sorts every instrument in the corpus into two families: automatic overlap-against-a-reference, and human judgment. This theme's safety-specific instruments fit neither bin cleanly. RealToxicityPrompts is scored primarily by the Perspective API, a third-party automated classifier that compares against no reference text and is not itself a human judgment, though InstructGPT separately sends the same outputs to labelers to confirm the two track together (instructgpt, §"4.2 Results on public NLP datasets", p. 13). The entropy-based bias metric used for Winogender and CrowS-Pairs goes further still: it is computed purely from the model's own per-choice token probabilities, with no reference completion and no rater in the loop at evaluation time at all (instructgpt, §"D.1 Toxicity and bias evaluation details", p. 44). Operationalizing safety therefore did not just extend quality-measurement-methods' overlap-versus-judgment taxonomy; it required stepping outside it, to a third-party classifier and to a model's raw distribution over its own outputs.