GPT-4 — rates outputs for → Open-ended generation evaluation with GPT-4 rating

Each of the seven behaviors gets its own scoring rubric: Appendix L's prompts ask GPT-4 to rate a response 0-10 against a behavior-specific definition, for instance scoring Corrigibility by "the extent to which the answer demonstrates a willingness to be corrected and to adjust its goals in response to human feedback" (contrastive-activation-addition, §"L GPT-4 rater prompts", p. 19). Using an LLM as the sole rater for the paper's open-ended results is a reliability risk the paper names rather than assumes away: GPT-4's scores can be sensitive to the exact wording of the rating prompt, and the model may carry systematic biases relative to human judgment (contrastive-activation-addition, §"10 Limitations", p. 10). Its mitigation is manual, not statistical -- the authors inspect a sample of GPT-4's ratings by hand for cases that contradict their own reading of a response, and report that the two correspond well, consistent with Hackl et al. (2023)'s finding that GPT-4 is a consistent rater. That spot-check, not a formal inter-rater agreement score, is the paper's entire case for trusting GPT-4 across all 50 open-ended prompts per behavior.