Llama 2 Chat
Steering Llama 2 via Contrastive Activation Addition — inherited
The dialogue-optimized, RLHF-finetuned variant of Llama 2, used as CAA's primary evaluation target (7B and 13B sizes) and compared against the Llama 2 base model to study how RLHF affects internal representations.
It's one thing to steer a raw language model's next-token predictions; it's another to steer a model that's already been trained through RLHF to behave a certain way in dialogue, where whatever CAA is manipulating internally has to coexist with everything that safety training already installed.
Llama 2 Chat's role as CAA's main evaluation target, at 7B and 13B scale, is set out ahead, along with its other job: the RLHF-tuned half of the comparison against the untuned base model, used to ask how much RLHF actually changed underneath.
The RLHF'd variant CAA steers
Llama 2 Chat is Llama 2's dialogue-optimized variant, produced by applying Reinforcement learning from human feedback (RLHF) on top of the pretrained base model to make it follow instructions and behave safely in a chat format (contrastive-activation-addition, §"3 Method", p. 3). It is the model every one of CAA's headline results is measured on, at both the 7B and 13B parameter scales: the Multiple-choice behavioral evaluation and layer sweep layer sweep, the Open-ended generation evaluation with GPT-4 rating ratings, the CAA combined with system-prompting comparison, and the CAA combined with supervised finetuning comparison all steer Llama 2 Chat rather than the base model.
Why steer a model already trained for safety
The choice is deliberate, not incidental. A steering method tested only on a raw pretrained model would leave open whether it still works once a model has already been shaped by RLHF to avoid the very behaviors — Sycophancy, excessive Refusal, Survival Instinct — the paper is trying to elicit or suppress. By steering Llama 2 Chat specifically, CAA is tested against, and on top of, safety training that is already trying to hold some of these behaviors down, a harder and more relevant test than steering an untrained model would be. That choice also sets up the paper's one experiment that looks inward rather than at outputs: the Base vs. RLHF-chat model representation comparison generates steering vectors from Llama 2 Chat's activations and compares them, layer by layer, against vectors from the untuned base model, finding elevated similarity specifically between layers 7 and 15 and significant transfer of base-derived vectors onto the chat model (contrastive-activation-addition, §"8.3 Comparing representations between base and chat models", p. 7). Because RLHF is the only thing separating the two models, that comparison reads as a direct measurement of what RLHF changed, and did not change, inside Llama 2 Chat.