Entropy-based bias metric
Training language models to follow instructions with human feedback — introduced
A bias measurement that converts a model's relative probability of producing each sentence in a Winogender/CrowS-Pairs pair into a binary distribution and reports its entropy in bits, where lower entropy indicates stronger (more biased) preference for one completion.
InstructGPT does not score its bias benchmarks the way their authors intended. Instead it asks how confidently the model prefers one sentence of a pair over the other, and reports that confidence as entropy in bits, lower entropy meaning a stronger, more biased preference.
The mathematics behind the recasting is worth working through slowly, since what it costs in comparability is easy to miss. Winogender and CrowS-Pairs, the two benchmarks it rescores, are the natural next stops.
How it fits
The entropy-based bias metric is InstructGPT's own instrument, introduced in this paper, for scoring both Winogender and CrowS-Pairs within Operationalizing truthfulness and social bias into measurable benchmarks — the theme collecting instruments purpose-built to catch harms that overlap-based capability metrics would miss. It converts a model's relative probability of producing each sentence in a paired-sentence item into a binary distribution and reports the entropy of that distribution in bits (instructgpt, §"D.1 Toxicity and bias evaluation details", p. 44).
The mathematics
For a paired item with two candidate completions, let $p$ be the model's normalized probability assigned to one completion (so $1-p$ is assigned to the other), computed from the two completions' relative likelihoods. The metric reports the binary (Shannon) entropy of this Bernoulli distribution,
$$H(p) = -p\log_2 p - (1-p)\log_2(1-p),$$
measured in bits. A model with no preference between the two paired sentences has $p = 0.5$, giving maximum entropy $H = 1$ bit; a model with a strong, decisive preference for one sentence over the other has $p$ near 0 or 1, giving entropy near 0 bits. Lower reported entropy is therefore interpreted as stronger — more biased — preference for one completion.
What the recasting costs
Applying this metric to existing benchmarks changes what gets measured, and the corpus documents this cost on both benchmarks it touches. It recasts the task of Winogender, which was designed as a coreference diagnostic where bias shows up as an accuracy asymmetry when only a pronoun's gender changes; the entropy recasting discards both correctness and direction, so a model that reliably resolves pronouns to the stereotypical referent and one that reliably resolves them to the anti-stereotypical referent produce identical low entropy and both read as "biased." InstructGPT's own interpretation concedes this: "the instructed models are more certain of their outputs regardless of whether or not their outputs exhibit stereotypical behavior" (instructgpt, §"4.2 Results on public NLP datasets", p. 14). It similarly strips the direction from CrowS-Pairs, whose native scoring is directional (how often a model favors the stereotyping sentence); under the entropy recasting, an intervention that flipped a model from consistently stereotypical to consistently anti-stereotypical would register as no change at all, or even as increased bias if the flip came with more confidence. The metric also sits outside the corpus's usual two-family taxonomy of automatic-overlap versus human-judgment instruments (Operationalizing truthfulness and social bias into measurable benchmarks — falls outside the taxonomy of → Automatic overlap scores versus human and preference judgments), since it needs no reference text and no rater at evaluation time at all, computed purely from the model's own output probabilities.