WSC (Winograd Schema Challenge)
Training language models to follow instructions with human feedback — inherited
A pronoun-coreference-resolution benchmark, part of SuperGLUE, included in the automatic evaluation suite.
The trophy does not fit in the suitcase because it is too big. What was too big? Winograd schemas hinge on pronouns like that, resolvable only through world knowledge, and WSC turns them into a benchmark.
After placing the task in InstructGPT's evaluation suite, the page turns to a wrinkle worth noticing: how the instance used here differs from its own parent benchmark, SuperGLUE.
How it fits
WSC, the Winograd Schema Challenge, is a pronoun-coreference-resolution benchmark drawn, alongside RTE, from SuperGLUE (Wang et al., 2019) into InstructGPT's public NLP benchmark suite. Each item presents a sentence containing an ambiguous pronoun whose correct referent depends on world knowledge or common-sense reasoning, and it predates InstructGPT as an established test of pronoun resolution (instructgpt, §"D Automatic evaluation details", p. 44; §"WSC (Winograd Schema Challenge)", p. 49).
What makes this instance different from its own benchmark
WSC is the clearest case in the corpus of a benchmark quietly measuring something other than what its name promises: it departs from the canonical format of SuperGLUE. SuperGLUE poses WSC as a binary judgment — "the pronoun 'she' refers to Joan, True or False?" — but InstructGPT instead asks the model to generate the pronoun's referent outright, then recovers a binary answer via a substring heuristic, checking whether the sampled completion contains the referent or vice versa (instructgpt, §"WSC (Winograd Schema Challenge)", p. 49). Two consequences follow directly from this. First, the reported accuracies are not comparable to the published SuperGLUE leaderboard, because both the task format and the answer-extraction rule differ from the canonical version. Second, the substring-containment check can misfire on verbose or hedged completions that happen to mention both candidate referents, a real risk given InstructGPT's own documented tendency toward Excessive hedging on simple questions. WSC is also, at just 104 questions, the smallest evaluation in the entire automatic suite, so its scores carry more sampling noise than any other benchmark in The public NLP benchmark suite used to catch regressions.