Steering doubles as interpretability

Members — a guided walk

1. Residual stream

   Every technique in this theme is a way of reading what is happening inside the residual stream: the running sum of layer outputs that a transformer's activations flow through and accumulate in as they move through the network. CAA already computes and injects its steering vectors into this stream at a chosen layer; the techniques that follow turn that same site into an object of study rather than just a place to intervene.

2. PCA visualization of contrastive activations

   PCA visualization of contrastive activations is the first such technique: projecting a behavior's contrast-pair dataset through Principal Component Analysis to check whether the model's internal representations separate along that behavior at all. It functions as a diagnostic run before committing to a full steering vector, a way of judging whether a given dataset will yield an effective one.

3. Principal Component Analysis (PCA)

   Principal Component Analysis, the linear dimensionality-reduction technique underneath that diagnostic, projects data onto the axes of maximal variance, implemented here via scikit-learn. Applied to residual stream activations on a contrastive dataset, it reduces high-dimensional internal representations down to a plane the authors can actually look at and judge for separation.

4. Behavioral clustering

   What that projection reveals, for datasets that work, is behavioral clustering: activations separating according to whether the model's answer matches the target behavior. The separation does not build up gradually across layers but emerges suddenly, around one-third of the way through the network, indicating that higher-level representations of the behavior form abruptly rather than accumulating token by token.

5. Letter clustering

   Letter clustering is the trivial separation the same projection always shows, whether or not a dataset captures anything real: activations split by which answer token, "A" or "B", they originate from, a pure artifact of the multiple-choice prompt format. The authors distinguish it carefully from behavioral clustering, since only the latter indicates the dataset is capturing the intended behavior rather than just its surface format.

6. Token-level cosine similarity analysis

   Token-level cosine similarity analysis reads the same vectors a different way: rather than projecting a dataset, it computes similarity between a finished steering vector and per-token activations during generation, showing the vector acts as a feature detector. It lights up at tokens like "I cannot help" for Refusal, or at the moment a model commits to a delayed reward for Myopic Reward, tracking how strongly the behavior is "present" token by token.

7. Inter-layer steering vector similarity and transfer

   Inter-layer similarity and transfer moves the analysis from within one layer to across depth: vectors extracted from nearby layers resemble each other closely, and that resemblance decays more slowly in the network's later half than its earlier half. A vector extracted at layer 13 still steers behavior when applied at other layers, evidence that the direction is a general representation of the behavior rather than an artifact specific to one layer.

8. Linear representation hypothesis

   The linear representation hypothesis is the theoretical claim that makes all of the above sensible in the first place: a formalization of the idea that high-level concepts are encoded as linear directions in an LLM's activation space. The paper cites it as part of the broader motivation for why activation-engineering techniques like steering vectors should work at all.

9. Linear representations of sentiment

   Linear representations of sentiment is the concrete prior instance backing that theory: earlier work verifying that sentiment specifically is linearly represented in LLM activations, and separately showing that Mean-Difference-derived vectors resemble PCA-derived ones. Between the hypothesis and this evidence, CAA's own interpretability results read less as a novel discovery and more as confirmation that behaviors, like sentiment, sit along linear directions too -- a confirmation the 2025 persona-vectors paper goes on to push further, from reading a direction whole to decomposing it into parts.

10. SAE decomposition of persona vectors

    SAE decomposition of persona vectors pushes the theme's doubling one step further: rather than only reading a persona vector as a single whole direction, it ranks a sparse autoencoder's decoder directions by cosine similarity to that vector, pulling out a small set of fine-grained, individually interpretable features underneath it. For 'evil,' top features correspond to insulting language, deliberate cruelty, and hacking-related content; for 'sycophancy,' to affirmative and promotional language; for 'hallucination,' to fictional world-building and fabricated detail, with each feature's causal role checked by steering with its own decoder direction.

11. Sparse autoencoders (SAEs)

    Sparse autoencoders are the unsupervised dictionary-learning technique that decomposition draws on: an approach that decomposes model activations into a larger set of sparse, individually interpretable feature directions. The paper trains BatchTopK SAEs on Qwen-2.5-7B-Chat's residual stream over a mix of pretraining, chat, and misalignment data, a second, unsupervised route to persona-relevant directions alongside the persona vector's own supervised extraction. The same architecture appears a second time in this theme under a different paper: Cunningham et al. (2023) train sparse autoencoders on the residual stream directly, and ask not just what directions can be pulled out of it but what its own coordinate basis means.

12. Privileged basis

    Privileged basis names the property of a coordinate system being unusually meaningful in its own right -- the standard basis immediately after an elementwise nonlinearity is one example, since each coordinate then corresponds to a single unit's output. The residual stream is not generally expected to have one: nothing about summing layer outputs picks out a preferred set of axes, so a priori any rotation of the space should be as meaningful as any other. Cunningham et al. (2023) find this expectation only partially holds, once outlier dimensions are taken into account.

13. Outlier dimensions

    Outlier dimensions are the exception: specific residual-stream coordinates that transformers write disproportionately large, information-bearing values to, a pattern Dettmers et al. (2022) attribute to the Adam optimizer storing gradients with finite precision in the residual basis. Cunningham et al. (2023) find this directly under their own apostrophe-feature case study: searching for the residual-stream dimension that most resembles the apostrophe-detecting dictionary feature turns up an outlier dimension, and the next several highest-activating dimensions by both sign are outlier dimensions too, not typical basis coordinates. The residual stream is read three ways across this theme -- as a site to steer and diagnose behavior in, as a vector decomposed into persona sub-features, and as a coordinate system whose basis is mostly arbitrary but not quite -- closing on the same fine print that keeps the default-basis baseline from being just another interchangeable rotation of the space.

Superthemes

• Decomposing the model into features
• Steering the model from inside: activation engineering and what it reveals

Edges explored within this theme

• Linear representation hypothesis — licenses the vector arithmetic of → Steering vector
• Linear representations of sentiment — cross validates against pca for → Mean Difference (MD) vector extraction
• PCA visualization of contrastive activations — screens datasets for → CAA steering vector construction (Mean Difference over multiple-choice contrast pairs)
• Letter clustering — is the confound for → Behavioral clustering
• Behavioral clustering — localizes suddenly in → Residual stream
• Principal Component Analysis (PCA) — is the projection technique behind → PCA visualization of contrastive activations
• Behavioral clustering — is evidence for → Linear representation hypothesis
• Token-level cosine similarity analysis — reveals as a feature detector → Steering vector
• Inter-layer steering vector similarity and transfer — demonstrates the generality of → CAA steering vector construction (Mean Difference over multiple-choice contrast pairs)
• Persona vector — admits a fine grained decomposition via → SAE decomposition of persona vectors
• SAE decomposition of persona vectors — applies the unsupervised machinery of → Sparse autoencoders (SAEs)
• Emergent misalignment — motivated the training data mix behind → SAE decomposition of persona vectors

Super edges touching this theme

• Steering doubles as interpretability — explains the one hyperparameter in → A behavior becomes a direction in activation space
• Base versus chat: what RLHF changes inside — repurposes the instruments of → Steering doubles as interpretability
• Steering doubles as interpretability — becomes an operational monitor in → Projection as early warning: reading the persona before it speaks hindsight
• Steering doubles as interpretability — surfaces artifacts explained by → Superposition and the case against the neuron basis

Sources

Every passage behind this theme’s member concepts.