Winogender
Training language models to follow instructions with human feedback — inherited
A benchmark dataset (Rudinger et al., 2018) of paired sentences used to probe gender bias in coreference-style completions, adapted here into a binary multiple-choice entropy evaluation.
Change nothing in a sentence but a pronoun and watch what a model assumes about who does what job. Winogender is built on exactly such minimal pairs, probing whether occupational stereotypes steer a model's reading of who a pronoun refers to.
The benchmark comes first, then a wrinkle: InstructGPT recasts it into an entropy measurement rather than using it as designed. That recasting produces a curious divergence from a toxicity benchmark scored the same way. Worth chasing down.
The benchmark
Winogender (Rudinger et al., 2018) is a benchmark of paired sentences built to probe occupational gender bias in coreference resolution: each pair holds a sentence fixed except for the gender of a pronoun referring to a person in an occupational role (e.g. a mechanic), testing whether a model's resolution tracks occupational gender stereotypes rather than the sentence's actual content. It predates InstructGPT by four years and is inherited here as one of the standard social-bias probes the paper reports on (instructgpt, §"4.2 Results on public NLP datasets", p. 14; §"Winogender", p. 45), consisting of 120 binary multiple-choice items in this evaluation.
Recast into an entropy measurement
InstructGPT does not use Winogender's native accuracy-based coreference scoring. As the Entropy-based bias metric — recasts the task of → Winogender edge explains, the paper instead converts each pair's completion log-probabilities into a Bernoulli distribution and reports its entropy in bits, where higher entropy signals less preference between the paired completions and is treated as less bias (instructgpt, §"Winogender", p. 45). This recasting discards both correctness and stereotype direction: a model that reliably resolves the stereotypical referent and one that reliably resolves the anti-stereotypical referent produce identical low entropy under this scoring, both read as "biased." The paper's own discussion concedes the slippage, noting that "the instructed models are more certain of their outputs regardless of whether or not their outputs exhibit stereotypical behavior" (instructgpt, §"4.2 Results on public NLP datasets", p. 14), so InstructGPT's finding of no significant improvement on Winogender is really a statement about model confidence under this metric, not directly about occupational-gender coreference errors.
Diverging from a toxicity benchmark under the same prompt
Winogender shares an exact instruction protocol with RealToxicityPrompts, the same basic, respectful, and biased prompt variants (instructgpt, §"RealToxicityPrompts", p. 45), which the RealToxicityPrompts — diverges under respectful prompting from → Winogender edge turns into a controlled comparison. Under the "respectful" instruction, RealToxicityPrompts toxicity falls, one of the paper's clean safety wins, while on Winogender the same instruction lowers entropy, which the paper's own metric reports as increased bias (instructgpt, §"4.2 Results on public NLP datasets", pp. 13-14). Part of this divergence traces to the entropy metric's confidence confound described above, but the paper's own framing survives it: an instruction that reduces measured toxicity can simultaneously appear to worsen a bias measure, evidence that safety benchmarks in this corpus do not collapse onto a single axis. Winogender is a member of the Operationalizing truthfulness and social bias into measurable benchmarks theme alongside CrowS-Pairs, TruthfulQA, and Perspective API-scored toxicity.