RTE (Recognizing Textual Entailment)
Training language models to follow instructions with human feedback — inherited
A binary textual-entailment benchmark, part of SuperGLUE, included in the automatic evaluation suite.
Given two sentences, does the first guarantee the second? That yes-or-no judgment, called textual entailment, is what RTE tests, and it is one of two tasks InstructGPT borrows from the SuperGLUE suite.
The task gets located within the regression-watching benchmark battery before the page describes what it actually asks and how it is used here. Since RTE belongs to SuperGLUE, that parent page is the natural place to go next.
How it fits
RTE, Recognizing Textual Entailment, is a binary classification task — given a premise and a hypothesis sentence, decide whether the premise entails the hypothesis — and one of the two tasks InstructGPT draws from SuperGLUE (Wang et al., 2019), a benchmark suite for general-purpose language understanding. Like the rest of the public NLP benchmark suite it belongs to, RTE predates this paper and is included purely as an evaluative check for the Alignment tax (instructgpt, §"D Automatic evaluation details", p. 44; §"RTE (Recognizing Textual Entailment)", p. 48).
What it is and its role here
RTE is scored as a multiple-choice completion, with the model's answer decided by comparing per-token log probabilities across the two candidate labels rather than by any overlap metric — the same mechanism used for HellaSwag, SST (Stanford Sentiment Treebank), and, in native SuperGLUE form, contrasted against the other SuperGLUE import in this corpus. That contrast is instructive: RTE was kept in its canonical multiple-choice format, while its sibling task WSC was not — WSC departs from SuperGLUE's canonical format entirely, generating free text instead of making a binary judgment (instructgpt, §"WSC (Winograd Schema Challenge)", p. 49). RTE is therefore the more straightforward of the two SuperGLUE-derived checks in the suite: its reported accuracy is directly comparable to published SuperGLUE RTE results in a way WSC's is not. It carries no dedicated edge of its own beyond its membership in the benchmark suite and its role as a fidelity contrast against WSC.