Llama 2
Steering Llama 2 via Contrastive Activation Addition — inherited
A family of pretrained and instruction/RLHF-finetuned language models (Touvron et al., 2023) ranging from 7B to 70B parameters. CAA is applied primarily to the 7B and 13B Chat variants, with the base model used for representation-comparison experiments.
Studying how RLHF changes a model's internals requires having both the RLHF'd model and the version that came before it, trained the same way apart from that last step. Llama 2 was a convenient choice for exactly that reason: a single family spanning 7 to 70 billion parameters, released with both raw pretrained weights and chat-tuned variants built on top of them.
Llama 2's role as CAA's testbed is set out ahead, the base and chat variants both put to use, one as the primary target for steering experiments, the other as the point of comparison for what RLHF actually changed.
An open family of pretrained and chat-tuned models
Llama 2 is Meta's family of language models (Touvron et al., 2023), released with weights available for research and commercial use at sizes from 7 to 70 billion parameters. Each size ships in two forms: a pretrained base model trained on a large corpus of publicly available text, and an instruction/RLHF-finetuned Chat variant optimized for dialogue and safety. The open release of both stages — base and chat, at multiple scales — is what makes Llama 2 usable as a research object rather than only as a product: unlike closed models such as GPT-4, its weights and intermediate activations are directly inspectable, a precondition for any method, including CAA, that works by reading and writing residual-stream activations rather than by prompting alone.
Two variants, one experiment design
CAA works with two of Llama 2's variants specifically, and the pairing is itself part of the experimental design. Llama 2 Chat at 7B and 13B is the primary target every steering, system-prompt, and finetuning result in the paper is measured on; the 7B base model appears once, to generate steering vectors independent of any RLHF tuning for the Base vs. RLHF-chat model representation comparison (contrastive-activation-addition, §"3 Method", p. 3). Because base and chat share weight initialization, pretraining data, and tokenizer, and differ only by the RLHF stage layered on top, any difference found between vectors extracted from the two is attributable to RLHF specifically rather than to some confound between unrelated models — a matched before-and-after design (contrastive-activation-addition, §"1 Introduction", p. 2) the paper could not run with a single-checkpoint model.
The 70B end of the family goes untested
Despite Llama 2 topping out at 70 billion parameters, none of CAA's experiments run at that scale — every result in the paper comes from the 7B or 13B checkpoints, smaller than the largest variant Touvron et al. released. That leaves the paper's central claims about layer optimality, transfer, and minimal capability cost formally untested at the size where a production deployment would be most likely to run. It also places CAA within a pattern common to the Activation engineering / representation engineering (technique family) literature it compares itself against: Turner et al.'s Activation Addition (ActAdd) was validated only on GPT-2-XL, smaller again than either Llama 2 size CAA tests, so demonstrating the technique on a genuinely RLHF-tuned, multi-billion-parameter open model was itself part of this paper's contribution.