SST (Stanford Sentiment Treebank)
Training language models to follow instructions with human feedback — inherited
A binary sentiment-classification benchmark (Socher et al., 2013) included in the automatic public NLP evaluation suite.
At bottom, SST asks a model one of the oldest questions in natural language processing: is this movie review positive or negative? It dates to 2013 and was annotated in unusual depth, with sentiment marked at every node of every parse tree.
The benchmark itself comes first, then its narrower role here as part of InstructGPT's automatic evaluation suite. More worth noting than the mechanics is how a decade-old task still earns its place in a modern alignment pipeline.
The benchmark
SST, the Stanford Sentiment Treebank (Socher et al., 2013), is a binary sentiment-classification dataset built from movie-review sentences, originally annotated at every node of each sentence's parse tree to support models (originally a recursive neural tensor network) that reason about sentiment compositionally rather than only at the sentence level. In this paper's evaluation, only sentence-level binary positive/negative labels are used. SST predates InstructGPT by nearly a decade and is inherited unmodified here as one of the standard public classification benchmarks in the automatic evaluation suite (instructgpt, §"D Automatic evaluation details", p. 44; §"SST (Stanford Sentiment Treebank)", p. 48), consisting of 872 binary multiple-choice items.
Its role here
SST is part of the The public NLP benchmark suite used to catch regressions theme, the battery of academic tasks InstructGPT runs against to catch the Alignment tax, general-capability regressions RLHF fine-tuning might introduce relative to plain GPT-3 (pretrained language model). Unlike DROP or translation, SST is one of the benchmarks where the tax proves mild: per Table 14, the 175B PPO-ptx model reaches 0.900 zero-shot accuracy against GPT-3's 0.898 (instructgpt, §"E.1 Performance on public NLP datasets", p. 56), essentially unchanged. Because SST is a simple multiple-choice classification task scored by likelihood rather than free generation, it belongs to the same cluster of "recoverable" benchmarks as HellaSwag, where the mechanical form of the task, ranking fixed options rather than producing open text, appears to make RLHF's capability cost easiest to pay back.