Behavioral clustering
Steering Llama 2 via Contrastive Activation Addition — introduced
The named phenomenon in which PCA-projected activations on a contrastive dataset separate according to whether the model's answer matches the target behavior, found to emerge suddenly around one-third of the way through a model's layers.
Somewhere around a third of the way through Llama 2's layers, something changes abruptly: activations that looked scattered and undifferentiated one layer earlier suddenly split cleanly into two groups, sorted by whether the model's answer matches the behavior being studied. That sudden transition is a clue about where abstract behavioral concepts actually take shape inside the network.
Traced across the seven tested behaviors, this pattern becomes part of the evidence for which layers are worth extracting a steering vector from in the first place.
Separation by behavior, not by prompt format
Behavioral clustering is CAA's name for the pattern PCA visualization of contrastive activations is designed to detect: a contrastive dataset's activations, projected onto their top two principal components, splitting into two groups according to whether the model's chosen answer matches the target behavior, rather than by some more superficial property of the prompt. The paper states the expectation directly: "for datasets truly capturing the behavior of interest, we expect the projections to also separate based on whether or not the model output matches that target behavior" (contrastive-activation-addition, §"3.2 Visualizing activations for contrastive dataset analysis", p. 4). It is explicitly distinguished from Letter clustering, the trivial split by answer token that the multiple-choice format guarantees regardless of whether any real behavioral signal is present.
A sudden onset around one-third of the way through
The clustering does not fade in gradually across the network's depth -- it appears abruptly at a specific layer. On the refusal dataset in Llama 2 7B Chat, layer 9 shows no separation by behavior; layer 10, one layer later, shows a clean split (Figure 2). Across the seven behaviors tested, this onset consistently falls "around one-third of the way through the layers" (contrastive-activation-addition, §"3.2 Visualizing activations for contrastive dataset analysis", p. 4), a pattern the authors link to prior work from Representation Engineering (Zou et al. 2023) showing emotion representations likewise emerging in middle and later layers rather than early ones. The Residual stream is therefore largely inert to a behavior's presence for the network's first third, then becomes linearly readable within the space of a single layer.
Direct evidence for linear representation
Behavioral clustering is not just background motivation for the Linear representation hypothesis -- it is a direct empirical instance of it. When residual-stream activations are projected onto only two principal components, a strictly linear operation, a high-level behavior like refusal becomes visibly separable, and does so suddenly rather than gradually. If the behavior's encoding were nonlinear or diffusely spread across many dimensions, a two-dimensional linear projection would have no particular reason to reveal a clean split. The suddenness is itself informative: it suggests the model computes the relevant abstraction at one point in depth and carries it forward largely unchanged, rather than gradually refining a distributed encoding -- the same layer range CAA's own layer sweeps for CAA steering vector construction (Mean Difference over multiple-choice contrast pairs) concentrate their steering effect in.