Representation Engineering (Zou et al. 2023)
Steering Llama 2 via Contrastive Activation Addition — inherited; Persona Vectors: Monitoring and Controlling Character Traits in Language Models — inherited
A prior body of techniques (Zou et al., 2023) for locating and extracting representations of high-level concepts such as honesty and emotion in LLMs, including testing the Mean Difference approach also used by CAA. CAA builds on this by focusing on steering rather than representation extraction.
Before anyone tried steering a model's outputs with these techniques, there was a more basic question: could you even find where a concept like honesty or a particular emotion lives inside a language model's activations at all? A body of work under the banner of representation engineering set out to answer exactly that, treating extraction as the goal rather than a means to steering.
CAA builds directly on this earlier program, sharing its use of the Mean Difference approach for extracting directions; where the two part ways is that CAA redirects that toolkit from representation extraction toward actually steering what a model says.
Reading representations before steering them
Representation Engineering (RepE), Zou et al.'s (2023) program, treats an LLM's activations as something to be read before it is something to be steered. The project develops several techniques for locating and extracting representations of high-level concepts -- honesty, emotion, and others -- inside a model's internal state, testing multiple extraction methods against each other, including the same Mean Difference procedure CAA later adopts wholesale. Steering appears in RepE's work, but as one downstream demonstration of what a located representation makes possible, not as the primary object of study. That ordering -- characterize the representation, then show it can be manipulated -- is the opposite emphasis from CAA, whose method section starts from steering and only afterward turns to what the resulting vectors reveal about internal structure.
Two changes: pair quality and goal
CAA names two specific ways it diverges from RepE. The first is about the data: "CAA employs an optimized multiple-choice format that results in more closely paired contrastive prompts that differ by only a single token" (contrastive-activation-addition, §"2 Related work", p. 2), tightening RepE's looser contrastive prompts into a Multiple-choice contrast-pair format where everything except the answer letter is held fixed. The second is about intent: CAA "build[s] on this work by focusing on steering rather than representation extraction, experimenting with a broader range of behaviors, and comparing steering to system-prompting and supervised finetuning" (contrastive-activation-addition, §"2 Related work", p. 2). The same class of Mean Difference direction is repurposed -- from a diagnostic to be read out, in RepE, to a lever to be pulled, in CAA, tested via CAA combined with system-prompting comparison and CAA combined with supervised finetuning comparison against System-prompting and Supervised fine-tuning (SFT).
An echo in CAA's own layer-wise findings
RepE resurfaces later in the paper, outside the related-work comparison, as independent support for a result CAA finds on its own. When CAA looks at where in the network a behavior becomes linearly readable via Behavioral clustering, it finds the separation emerges "around one-third of the way through the layers," and notes this "aligns with past work showing emotion representations emerge in middle and later layers (Zou et al., 2023)" (contrastive-activation-addition, §"3.2 Visualizing activations for contrastive dataset analysis", p. 4). The paper CAA distinguishes itself from methodologically turns out to have already observed the same depth-dependent pattern for a different concept, strengthening the claim that middle-layer emergence is a general property of how LLMs represent high-level behavior.