Automatic overlap scores versus human and preference judgments
Deciding, with a number, that one model's answer beats another's turns out to need more than one kind of instrument. This theme covers automatic overlap scores like ROUGE-L and BLEU, which compare text against a fixed reference, judgment-based metrics like win rate and Elo score, which ask a human to simply prefer one output over another, and a third approach a 2023 paper adds, where the judge is a language model instead of a person.
The families sit in contrast here, and inter-labeler agreement runs underneath the human-judgment ones, since every preference number is only as trustworthy as the people producing it; the 2023 paper's model-judged scores get their own version of that check, a manual spot-comparison against the ratings a person would have given.
Both InstructGPT and Constitutional AI need to say, quantitatively, whether one model's output is better than another's, and this theme gathers the two families of measurement the corpus uses to do that, plus a third that Steering Llama 2 via Contrastive Activation Addition (2023) adds. Automatic overlap metrics compare generated text against a reference: ROUGE-L for summarization, BLEU for translation, and F1 score for span-overlap question answering, all cheap to compute but blind to whether an answer is actually preferred. Judgment-based metrics instead ask an evaluator to compare or rate outputs directly: InstructGPT's win rate reports how often labelers prefer one model's output over a fixed baseline, and its Likert quality rating scores a single output on a 1-7 scale; Constitutional AI's Elo score performs the analogous role from crowdworker pairwise comparisons, while its absolute harmfulness score rates a single conversation on a 0-4 scale via a model trained on crowdworker ratings. Inter-labeler agreement rate underlies all of the judgment-based measures, since a preference metric is only as trustworthy as the consistency of the humans producing it. Open-ended generation evaluation is a third answer to the same underlying problem, judgment-based but with the human rater replaced: CAA applies steering vectors to free-form answers rather than multiple-choice questions, then has GPT-4 score each response 1-10 for how strongly it displays the targeted behavior, using a behavior-specific rubric prompt, with a manual spot-check against human ratings standing in for the inter-annotator-agreement check a human-only pipeline would run. Together, overlap scores, human judgment, and LLM judgment are this corpus's three answers to the same problem: reference text rarely captures what a preference actually is.