WSC (Winograd Schema Challenge) — departs from the canonical format of → SuperGLUE

WSC is credited to SuperGLUE, but the task InstructGPT actually runs is not the one on the SuperGLUE leaderboard. SuperGLUE poses WSC as a binary judgment — "the pronoun she refers to Joan, True or False?" — while InstructGPT asks the model to generate the pronoun's referent outright and then recovers a binary answer with a substring heuristic: it checks whether the sampled completion contains the referent or vice versa, answering True if so and False otherwise (instructgpt, §"WSC (Winograd Schema Challenge)", p. 49). Two things follow. First, the reported accuracies are not comparable to published SuperGLUE WSC results, because both the task format and the answer-extraction rule differ. Second, the contains-check can misfire on verbose or hedged completions that mention both candidate referents. RTE, the other task imported from the same suite, was kept in its native multiple-choice form — so of the two SuperGLUE imports, one measures the model's judgments directly while the other measures its generations filtered through string matching, on a sample of just 104 questions, the smallest evaluation in the entire automatic suite.