QuAC (Question Answering in Context)
Training language models to follow instructions with human feedback — inherited
A conversational question-answering benchmark (Choi et al., 2018) evaluated in the paper's automatic evaluation suite.
Questions in a conversation lean on what was already asked; a bare follow-up only makes sense in context. QuAC tests exactly that, posing sequences of questions about a passage where each answer depends on the dialogue so far.
Folded into the eight-task suite InstructGPT runs to catch regressions, it carries no connections of its own among the benchmarks, one of the few pages here that stands entirely alone.
How it fits
QuAC, Question Answering in Context (Choi et al., 2018), is a conversational question-answering benchmark: unlike single-turn QA datasets, it requires answering a sequence of questions about a passage where later questions depend on the conversational context established by earlier ones. It predates InstructGPT and enters the paper as one of the eight tasks in the public NLP benchmark suite used to detect the Alignment tax (instructgpt, §"D Automatic evaluation details", p. 44; §"QuAC (Question Answering in Context)", p. 47).
What it is and its role here
QuAC is scored, like DROP and SQuAD v2, as a free-response span-extraction task via F1 against a target completion, which places it among the three benchmarks in the suite that the corpus's overlap-metric family actually instruments, as opposed to the remaining four tasks, HellaSwag, SST (Stanford Sentiment Treebank), RTE (Recognizing Textual Entailment), and WSC (Winograd Schema Challenge), which are instead scored by comparing the model's own per-token log probabilities across multiple-choice options with no reference text at all (instructgpt, §"D Automatic evaluation details", p. 44). QuAC carries no dedicated edge of its own in this graph — no ablation or mitigation experiment singles it out the way DROP and SQuAD v2 were promoted to tuning targets — which means it functioned in InstructGPT purely as one more member of the evaluation battery: a check, alongside the rest of the suite, for whether RLHF fine-tuning traded away conversational comprehension ability for its gains in preference-based instruction following.