Entropy-based bias metric — strips the direction from → CrowS-Pairs

CrowS-Pairs' native scoring is directional: it reports how often a model assigns higher likelihood to the stereotyping sentence of a pair, with 50% as the unbiased ideal. InstructGPT instead scores it "using entropy over multiple choice probabilities, the same as was done for Winogender" (instructgpt, §"CrowS-Pairs", p. 45), keeping only how strongly the model prefers one sentence and discarding which one. Under this scoring, an intervention that flipped a model from consistently stereotypical to consistently anti-stereotypical would register as zero change — or as increased bias, if the flip came with more confidence. The consequence is visible in Table 14: the "respectful" instruction drives 175B PPO-ptx entropy on CrowS-Pairs from 0.413 down to 0.243 (instructgpt, §"E.1 Performance on public NLP datasets", p. 56), which the metric reports as sharply increased bias, when the more parsimonious mechanism is that instruction-following makes the model more decisive about everything. The paper's negative bias result is therefore partly an artifact of a measurement that cannot tell prejudice from confidence.