PCA visualization of contrastive activations
Steering Llama 2 via Contrastive Activation Addition — introduced
A diagnostic technique projecting model activations on a contrastive dataset via PCA to assess whether that dataset's activations linearly separate along the target behavior, used to judge whether a dataset will yield an effective steering vector.
Not every dataset of contrastive examples turns into a useful steering vector, and there's no way to tell in advance just by reading the questions. Projecting activations onto their two directions of greatest variance offers a shortcut: a dataset that genuinely captures the target behavior will visibly split into two clouds when plotted this way; one that doesn't, won't.
From that rough sanity check, the story moves to a sharper finding: the two clusters this technique reveals aren't stable across layers, they snap into separation abruptly at a particular depth in the network, a pattern picked up again under behavioral clustering.
A screening step before vector construction
PCA visualization of contrastive activations is CAA's own diagnostic, introduced to answer a question that comes before "does this steering vector work": does this dataset's activations even separate along the target behavior in the first place? The technique projects a contrastive dataset's residual-stream activations, taken at the token position of the answer letter, onto the top two components found by Principal Component Analysis (PCA) via Scikit-learn, at each layer of the model, and checks for separation by eye. The authors describe this as useful "for determining whether a dataset will enable the generation of effective steering vectors" (contrastive-activation-addition, §"3.2 Visualizing activations for contrastive dataset analysis", p. 4), building on the lead author's own earlier blog post on visualizing sycophancy datasets (Panickssery, 2023b).
Two kinds of separation, only one meaningful
Because every prompt pair ends in the token "A" or "B", a projection at that position will always show some split -- Letter clustering, a pure artifact of the prompt format. What the visualization is actually looking for is Behavioral clustering: separation by whether the model's chosen answer matches the target behavior, not merely by which letter it is. Figure 2 in the paper shows the difference concretely on the refusal dataset in Llama 2 7B Chat: at layer 9 there is no visible behavioral split, and one layer later, at layer 10, "the visible behavioral clustering emerges suddenly at layer 10" (contrastive-activation-addition, §"3.2 Visualizing activations for contrastive dataset analysis", p. 4).
Screening precedes, and justifies, construction
This screening step logically precedes CAA steering vector construction (Mean Difference over multiple-choice contrast pairs)'s Mean Difference procedure, which averages an activation difference over an entire dataset at a chosen layer. That average is only meaningful once behavioral clustering confirms the representation being differenced actually tracks the intended behavior rather than the prompt's surface format; a dataset showing only letter clustering, with no behavioral split, is a warning sign that the resulting vector may encode the wrong thing. That the separation is visible at all in a plain two-dimensional, unsupervised projection is also part of the paper's broader evidence, alongside the Linear representation hypothesis, that these behavioral representations are simple linear directions rather than more complex, entangled features.