CrowS-Pairs
Training language models to follow instructions with human feedback — inherited
A benchmark dataset (Nangia et al., 2020) of paired sentences designed to measure social bias (stereotype vs. anti-stereotype) in language models.
One sentence leans on a social stereotype; its twin differs by only a few words and does not. CrowS-Pairs assembles many such pairs, across race, gender, religion, and more, to check which version a language model finds more natural.
The benchmark stands on its own terms first, then gets rescored through InstructGPT's entropy-based bias metric, the one connection that matters here, since the recasting does the real work, not the pairs themselves.
The benchmark
CrowS-Pairs (Crowdsourced Stereotype Pairs) is a dataset introduced by Nangia et al. (2020) consisting of pairs of minimally different sentences, one invoking a social stereotype about a historically disadvantaged group, one an anti-stereotypical or unrelated variant, spanning categories such as race, gender, religion, and disability. Its original scoring method is directional: it measures how often a masked-language-model scoring procedure assigns higher likelihood to the stereotyping sentence than to its paired alternative, with 50% representing an unbiased model. CrowS-Pairs predates InstructGPT and is inherited here as one of the public bias probes InstructGPT is run against (instructgpt, §"4.2 Results on public NLP datasets", p. 14; §"CrowS-Pairs", p. 45), containing 1,508 sentence pairs in this evaluation.
Recast through InstructGPT's entropy metric
InstructGPT does not use CrowS-Pairs' native directional scoring. As the Entropy-based bias metric — strips the direction from → CrowS-Pairs edge explains, the paper instead scores it "using entropy over multiple choice probabilities, the same as was done for Winogender" (instructgpt, §"CrowS-Pairs", p. 45), which keeps only how strongly the model prefers one sentence in a pair and discards which one it prefers. Under this recasting, a model that flipped from confidently stereotypical to confidently anti-stereotypical would register no improvement, or even an apparent increase in bias if the flip came with more confidence. This is not a hypothetical concern: the "respectful" instruction drives the 175B PPO-ptx model's entropy on CrowS-Pairs from 0.413 down to 0.243 (instructgpt, §"E.1 Performance on public NLP datasets", p. 56), which the paper's own metric reports as sharply increased bias, when a more parsimonious explanation is that instruction-following simply makes the model more decisive across the board. CrowS-Pairs sits, along with Winogender and Perspective API-scored toxicity, in the Operationalizing truthfulness and social bias into measurable benchmarks theme built to catch harms that overlap-based capability metrics would miss.