Open-ended generation evaluation with GPT-4 rating
Steering Llama 2 via Contrastive Activation Addition — introduced
The paper's protocol for testing whether CAA generalizes beyond multiple-choice settings: steering vectors are applied while generating free-form answers to open-ended questions, and GPT-4 rates each response 1-10 for how strongly it displays the targeted behavior using behavior-specific rubric prompts.
A steering vector that only works on artificial multiple-choice questions wouldn't tell you much about whether it actually shifts how a model behaves in the wild. The real test is free-form generation: no answer options to nudge toward, just an open question and whatever the model chooses to say once the vector is added.
How those open-ended answers get scored comes next: GPT-4 rating each response on a 1-10 scale for how strongly it displays the targeted behavior, and the check against a human reader that the authors ran before trusting that score.
From contrived questions to free text
A steering method that only worked on contrived multiple-choice questions would be a narrow result. Open-ended generation evaluation is CAA's test of whether the effect generalizes: steering vectors are applied at generation time to free-form answers rather than to a fixed answer letter, using questions either manually written — for Sycophancy, to probe a broader range of sycophancy-relevant scenarios — or adapted from the held-out Multiple-choice behavioral evaluation and layer sweep set by stripping the answer options and keeping only the question (contrastive-activation-addition, §"4.2 Open-ended generation", p. 4). Appendix G's transcripts show the resulting range: adding the Sycophancy vector turns a question about an all-dessert diet into enthusiastic agreement, subtracting it into a health-conscious refusal (contrastive-activation-addition, §"G Examples of open-ended generation with CAA", p. 16).
GPT-4 grades against a per-behavior rubric
Each response is scored 0–10 by GPT-4 against a rubric written specifically for that behavior — Appendix L's Corrigibility prompt, for instance, asks for a score reflecting "the extent to which the answer demonstrates a willingness to be corrected" (contrastive-activation-addition, §"L GPT-4 rater prompts", p. 19). Using an LLM as the sole rater is a named risk, not an assumed-away one: the paper's Limitations section flags that GPT-4's scores can be sensitive to rubric wording and may carry systematic bias relative to human judgment, and its mitigation is a manual spot-check against the authors' own reading rather than a formal inter-rater statistic, reported to correspond well and consistent with Hackl et al. (2023)'s finding that GPT-4 is a reliable rater (contrastive-activation-addition, §"10 Limitations", p. 10).
Where it disagrees with the multiple-choice score
Run alongside Multiple-choice behavioral evaluation and layer sweep on the same finetuning-versus-CAA comparison, the two regimes locate CAA's marginal value differently: finetuning alone often reaches near-ceiling multiple-choice scores, leaving little visible room for CAA to add, while the same checkpoints score far from ceiling under GPT-4's 0–10 scale, where CAA's addition moves the number by a larger margin (contrastive-activation-addition, §"6 Comparison to finetuning", p. 6). Substituting a capable model for a human judge is also the move Constitutional AI's RL from AI Feedback (RLAIF) makes one step earlier in a training pipeline, closing the same labeling-at-scale bottleneck this paper faces at evaluation time instead.