Letter clustering
Steering Llama 2 via Contrastive Activation Addition — introduced
The named phenomenon in which PCA-projected activations on the contrastive multiple-choice dataset separate merely by which answer token ("A" or "B") they originate from -- a trivial artifact of the prompt format that the authors distinguish from the more meaningful behavioral clustering.
Project a dataset's activations with PCA and a split will appear almost no matter what, because the multiple-choice format itself guarantees one: every prompt ends in either the letter A or B, and that alone is enough to separate the activations into two groups. The question is whether that split means anything.
Separating that superficial artifact from the more meaningful behavioral clustering it can be confused with is the point of what follows, along with why the two get treated as distinct signals when judging whether a contrastive dataset is any good.
The split you get for free
Letter clustering is the separation that shows up in a PCA projection of CAA's contrastive-dataset activations no matter what the dataset is about. Because every prompt pair in the Multiple-choice contrast-pair format ends with the answer token "A" or "B", and activations are extracted at exactly that token position, projecting them onto two principal components will always split the data into two groups corresponding to which letter is present: "activations can always be separated based on which token ('A' or 'B') they originate from ('letter clustering')" (contrastive-activation-addition, §"3.2 Visualizing activations for contrastive dataset analysis", p. 4). The paper is careful to name this and set it aside as the null result to expect, not the signal being searched for.
The confound behavioral clustering has to be checked against
The reason letter clustering needs its own name, rather than being a footnote, is that CAA steering vector construction (Mean Difference over multiple-choice contrast pairs) differences activations at exactly this same answer-letter position when it builds a steering vector. Without a separate check for Behavioral clustering -- separation by whether the chosen answer matches the target behavior, not merely by which letter it is -- there would be no way to tell whether a resulting Mean Difference vector encodes the intended behavior or simply the distinction between the tokens "A" and "B". A dataset can show letter clustering alone, with no behavioral split at all, which is exactly the failure mode PCA visualization of contrastive activations is meant to catch before a vector is built and used.
A consequence of the format, not the behavior
Letter clustering would appear in a PCA projection of any dataset built in the Multiple-choice contrast-pair format, not just CAA's seven behavioral datasets -- it reflects the tokenizer's encoding of "A" versus "B" at the answer position, which has nothing to do with sycophancy, refusal, or any other target behavior. That is precisely why it functions as a useful negative control: a split that always fires regardless of dataset content sets a baseline against which the presence or absence of behavioral clustering can be judged. If a new dataset showed only letter clustering and no behavioral split at any layer, that would be evidence the dataset fails to elicit the intended behavior distinction, rather than evidence about the model's representations.