Steering Llama 2 via Contrastive Activation Addition
arXiv:2312.06681 · PDF · 19 pp. · its place among the papers
One paper, several tellings. The big picture states the paper’s own thesis and contributions in its own terms; Inside the paper follows the paper’s own arc; Across the corpus traces how it connects to the other papers; The concepts lists everything the paper uses, in reading order. Pick a tab, then use the +/− toggles to open it level by level, or set a granularity to read the whole thing at that zoom.
Every alignment technique on offer — RLHF, instruction finetuning, prompt engineering — works either by updating a model's weights or by hoping a prompt lands, and none of them says where inside the network a behavior like refusal or sycophancy actually lives. The paper's own claim is narrower and sharper than that: a behavior is a direction in the model's activations, and that direction can be found, dosed, and reversed without touching a single weight.
What follows is the paper's argument in its own vocabulary, at full altitude: the thesis that a behavior is a dosable direction, the construction that turns contrast pairs into one, the evidence across seven behaviors and two evaluation formats, what the method reveals about representation and about RLHF itself, and the limits the paper reports against its own result.
Read at
The 2023 argument: a behavior is a direction you can dose (5 chapters)
Evidence for the thesis runs on three legs: multiple-choice and open-ended evaluations showing all seven behaviors move in the intended direction; a capability floor, MMLU, that barely moves under the same intervention; and an interpretability chapter that reads the same vectors as measurements of where behavior lives and what RLHF changed. The paper reports its own limits alongside the result — an unoptimized finetuning baseline, a norm choice it flags against itself, GPT-4 rater noise — and closes by offering the same machinery as a red-teaming tool.
Chapter 1The thesis: a behavior is a direction you can dose (4 concepts)
The paper's central claim, stated plainly in its introduction and defended for the rest of its length, is that a high-level behavior — sycophancy, refusal, corrigibility — corresponds to a single direction in a model's activation space, discoverable from a dataset of paired prompts and controllable by adding that direction back into the residual stream with a signed multiplier. The premise licensing this is the linear representation hypothesis: that concepts this abstract still sit along linear directions rather than requiring some more tangled representation to describe them.
One direction, one behavior concept (4 connections)
A steering vector is a single direction in a model's activation space encoding a target behavior, such that adding it to a forward pass shifts output toward the behavior, and subtracting it shifts output away — the object the entire thesis rests on.
connections
- Linear representation hypothesis — licenses the vector arithmetic of → Steering vector
- Persona vector — specializes for character traits → Steering vector
- Steering multiplier — sets the dose for → Steering vector
- Token-level cosine similarity analysis — reveals as a feature detector → Steering vector
The method the thesis is built to test concept (12 connections)
Contrastive Activation Addition is the paper's name for the specific method of building and applying a steering vector: extract it from contrast pairs, add it to the residual stream at every post-prompt token position, evaluated across seven behaviors on Llama 2 Chat.
connections
- Automated persona vector extraction pipeline — automates away the bespoke curation of → Contrastive Activation Addition (CAA) hindsight
- CAA as an adversarial red-teaming tool (proposed application) — inverts the purpose of → Contrastive Activation Addition (CAA)
- CAA combined with supervised finetuning comparison — isolates a generalization advantage for → Contrastive Activation Addition (CAA)
- In-Context Vectors (ICV) — spans every layer unlike → Contrastive Activation Addition (CAA)
- Contrastive Activation Addition (CAA) — skips the head search required by → Inference-Time Intervention (ITI)
- MMLU (Massive Multitask Language Understanding) — sets the capability floor for → Contrastive Activation Addition (CAA)
- Multiple-choice behavioral evaluation and layer sweep — supplies the layer hyperparameter for → Contrastive Activation Addition (CAA)
- Post-hoc (inference-time) steering mitigation — reinstantiates → Contrastive Activation Addition (CAA) hindsight
- Contrastive Activation Addition (CAA) — operates on top of → Reinforcement learning from human feedback (RLHF) hindsight
- Contrastive Activation Addition (CAA) — repurposes for steering relative to → Representation Engineering (Zou et al. 2023)
- Residual stream — is the injection site whose growth constrains → Contrastive Activation Addition (CAA)
- Steering at targeted token positions (proposed extension) — would relax the quality ceiling of → Contrastive Activation Addition (CAA)
Why a vector should be enough concept (4 connections)
The linear representation hypothesis holds that high-level concepts are encoded as linear directions in an LLM's activation space — the theoretical premise that makes it plausible for a single added vector to shift a whole behavior rather than produce noise.
connections
- Activation engineering / representation engineering (technique family) — empirically motivated the formalization of → Linear representation hypothesis
- Behavioral clustering — is evidence for → Linear representation hypothesis
- Persona vector — quantifies empirical support for → Linear representation hypothesis
- Linear representation hypothesis — licenses the vector arithmetic of → Steering vector
Dosed, not just switched concept (3 connections)
The steering multiplier is what turns a direction into a controllable intervention: sign selects whether the behavior is encouraged or suppressed, and magnitude sets how strongly, though the usable range is capped by text-quality degradation at high magnitudes.
connections
- Persona vector — extends the dosing role of into training → Steering multiplier
- Steering multiplier — sets the dose for → Steering vector
- Steering vector normalization choices — makes cross behavior comparable for → Steering multiplier
Chapter 2The construction: contrast pairs to a vector (4 concepts)
Building the vector takes three decisions, stated in Section 3. Contrast pairs are multiple-choice questions differing by exactly one token — the answer letter — so nothing else can leak into the difference. The Mean Difference method averages that difference across a whole dataset rather than trusting any single pair. And the resulting vector is injected into the residual stream, the running sum every layer writes into, at every token position after the prompt, so its effect compounds across the entire generated response.
A single token separates positive from negative concept (2 connections)
Every contrast pair is a multiple-choice question ending in answer letter A or B, with the positive and negative prompt otherwise identical — a design meant to cancel every confound except the one variable the paper wants to isolate.
connections
- Multiple-choice contrast-pair format — cancels the confounders in → CAA steering vector construction (Mean Difference over multiple-choice contrast pairs)
- Multiple-choice contrast-pair format — repurposes the ab mechanic of → Multiple-Choice Comparison Format hindsight
The averaging step concept (3 connections)
Mean Difference extraction averages the activation difference between positive and negative completions across an entire dataset of pairs, rather than trusting the noise in any single comparison — the step that turns hundreds of individually noisy differences into one usable direction.
connections
- Mean Difference (MD) vector extraction — supplies the extraction rule for → CAA steering vector construction (Mean Difference over multiple-choice contrast pairs)
- Linear representations of sentiment — cross validates against pca for → Mean Difference (MD) vector extraction
- Response-conditioned contrastive vector computation — supplies manufactured contrast sets to → Mean Difference (MD) vector extraction
Equation 1, in full concept (6 connections)
Formalized as a single equation: for a dataset of (prompt, positive completion, negative completion) triples, take the layer-L activation difference at the answer-letter position for each triple, and average across the dataset — the complete recipe for a CAA steering vector.
connections
- CAA steering vector construction (Mean Difference over multiple-choice contrast pairs) — refines the single pair recipe of → Activation Addition (ActAdd)
- Inter-layer steering vector similarity and transfer — demonstrates the generality of → CAA steering vector construction (Mean Difference over multiple-choice contrast pairs)
- Mean Difference (MD) vector extraction — supplies the extraction rule for → CAA steering vector construction (Mean Difference over multiple-choice contrast pairs)
- Multiple-choice contrast-pair format — cancels the confounders in → CAA steering vector construction (Mean Difference over multiple-choice contrast pairs)
- PCA visualization of contrastive activations — screens datasets for → CAA steering vector construction (Mean Difference over multiple-choice contrast pairs)
- Response-conditioned contrastive vector computation — generalizes beyond the answer letter recipe of → CAA steering vector construction (Mean Difference over multiple-choice contrast pairs) hindsight
Where the vector is added back in concept (4 connections)
The residual stream is the running sum of every layer's output that a transformer's computation flows through and accumulates in — the site CAA chooses to inject its vector into, rather than attention activations or a searched set of attention heads.
connections
- Behavioral clustering — localizes suddenly in → Residual stream
- Residual stream — is the injection site whose growth constrains → Contrastive Activation Addition (CAA)
- Response-conditioned contrastive vector computation — sweeps and selects a layer within → Residual stream
- Steering outside the residual stream (proposed extension) — would localize representations beyond → Residual stream
Chapter 3The evidence: seven behaviors move, generalization holds (4 concepts)
Two evaluation regimes carry the paper's evidence. Multiple-choice behavioral evaluation sweeps every layer at fixed multipliers and finds all seven tested behaviors move in the intended direction, with a consistent optimal layer across behaviors. Open-ended generation evaluation repeats the test in free text, scored by GPT-4 against a rubric written per behavior, and the effect survives leaving the contrived multiple-choice format. A third piece of evidence is negative: MMLU, run under every steering condition, shows no significant change, which the paper reads as its capability floor holding.
Every one of seven, in the intended direction concept (2 connections)
Multiple-choice behavioral evaluation sweeps a steering vector across every layer at multipliers of -1 and +1, scoring effect as the average token probability given to the behavior-matching answer on 50 held-out questions — and finds a consistent optimal layer where every tested behavior moves as intended.
connections
- Multiple-choice behavioral evaluation and layer sweep — supplies the layer hyperparameter for → Contrastive Activation Addition (CAA)
- Open-ended generation evaluation with GPT-4 rating — shows a different margin than → Multiple-choice behavioral evaluation and layer sweep
The result survives leaving multiple-choice concept (4 connections)
Open-ended generation evaluation applies the same steering vectors to free-form answers rather than contrived A/B questions, testing whether the effect generalizes beyond the format it was extracted from — necessary, the paper argues, for the method to be useful at all.
connections
- GPT-4 — rates outputs for → Open-ended generation evaluation with GPT-4 rating
- Open-ended generation evaluation with GPT-4 rating — shows a different margin than → Multiple-choice behavioral evaluation and layer sweep
- Open-ended generation evaluation with GPT-4 rating — shares the ai judge move with → RL from AI Feedback (RLAIF) hindsight
- Trait expression score (LLM judge) — succeeds the sampled rating protocol of → Open-ended generation evaluation with GPT-4 rating hindsight
The instrument doing the scoring concept (2 connections)
GPT-4 rates every open-ended response 1-10 against a rubric written per behavior, standing in for a human evaluator at a scale hand-scoring could not match.
connections
- GPT-4 — authors → Custom refusal contrastive dataset
- GPT-4 — rates outputs for → Open-ended generation evaluation with GPT-4 rating
And general capability barely moves concept (4 connections)
MMLU, 57 subjects of multiple-choice general knowledge, is run under every steering condition unmodified, and the paper reports no significant change to the model's average probability of the correct answer — the negative result that lets the positive results stand without an asterisk.
connections
- MMLU (Massive Multitask Language Understanding) — reruns the question behind → Alignment tax hindsight
- MMLU (Massive Multitask Language Understanding) — sets the capability floor for → Contrastive Activation Addition (CAA)
- Post-hoc (inference-time) steering mitigation — is checked against → MMLU (Massive Multitask Language Understanding)
- MMLU (Massive Multitask Language Understanding) — tests a different competence than → SuperGLUE hindsight
Chapter 4What it reveals: emergence and what RLHF changed (4 concepts)
Beyond controlling behavior, the paper's own techniques double as findings about the model. Behavioral clustering shows a targeted behavior separating linearly in activation space suddenly, around one-third of the way through the network's layers, rather than emerging gradually — direct evidence for the linear representation hypothesis the method assumes. Token-level cosine similarity turns the same vector into a passive reader, lighting up at semantically relevant tokens with no intervention at all. And comparing vectors built from Llama 2's base and RLHF-tuned chat variants finds them unusually similar in the middle layers, which the paper reads as a measurement of RLHF itself: it reweights which behaviors get expressed without much rewriting how they're represented internally.
Behavior separates suddenly, not gradually concept (3 connections)
Behavioral clustering is the finding that a contrastive dataset's activations separate by whether the model's answer matches the target behavior, and that this separation appears abruptly around one-third of the way through the network's layers rather than building up gradually.
connections
- Letter clustering — is the confound for → Behavioral clustering
- Behavioral clustering — is evidence for → Linear representation hypothesis
- Behavioral clustering — localizes suddenly in → Residual stream
A passive detector for free concept (1 connection)
Token-level cosine similarity between a steering vector and ordinary per-token activations, computed with no intervention at all, tracks how strongly the target behavior is present at each token — the same object built to control turns out to also detect.
connections
- Token-level cosine similarity analysis — reveals as a feature detector → Steering vector
What one training stage did and didn't do concept (2 connections)
Comparing steering vectors generated from Llama 2's base model against Llama 2 Chat finds elevated similarity specifically between layers 7 and 15, and finds base-derived vectors still steer the chat model — evidence that RLHF changes which behaviors surface more than it changes how they're represented.
connections
- Finetuning shift (activation-shift metric) — generalizes the single snapshot logic of → Base vs. RLHF-chat model representation comparison hindsight
- Base vs. RLHF-chat model representation comparison — measures the effect of → Reinforcement learning from human feedback (RLHF) hindsight
The stage being read, not modified concept (6 connections)
RLHF is the training paradigm that produced Llama 2 Chat from Llama 2 base; the paper never modifies it, only uses the base/chat pair it produced as an instrument for reading what that one training stage left behind internally.
connections
- Asynchronous Reward-Learning Architecture — gives concrete shape to → Reinforcement learning from human feedback (RLHF)
- Base vs. RLHF-chat model representation comparison — measures the effect of → Reinforcement learning from human feedback (RLHF) hindsight
- Contrastive Activation Addition (CAA) — operates on top of → Reinforcement learning from human feedback (RLHF) hindsight
- Reward model (RM) — anchors the reward signal of → Reinforcement learning from human feedback (RLHF)
- Reinforcement learning from human feedback (RLHF) — swaps the labeler in → RL from AI Feedback (RLAIF)
- Sycophancy — misgeneralizes the training objective of → Reinforcement learning from human feedback (RLHF) hindsight
Chapter 5The limits and the offer (4 concepts)
The paper reports its own weak points rather than leaving them implicit. The comparison to finetuning uses one fixed, unoptimized set of hyperparameters, so the finetuning baseline may understate what that lever can do. The comparison to system-prompting likewise searches a limited set of prompts. A vector-normalization choice — standardizing magnitude across behaviors but not across layers — is flagged as a possible confound in the paper's own layer-optimality results. Against these limits, the paper closes by offering its method a new job: using CAA as an adversarial red-teaming tool, deliberately eliciting an unwanted behavior to test whether training actually removed it or only suppressed it.
An unoptimized baseline concept (2 connections)
The finetuning comparison uses one fixed set of hyperparameters — one epoch, plain SGD, no tuning of learning rate or loss function — which the paper itself flags as likely understating what a better-optimized finetuning baseline could achieve.
connections
- CAA combined with supervised finetuning comparison — isolates a generalization advantage for → Contrastive Activation Addition (CAA)
- CAA combined with supervised finetuning comparison — runs head to head against → Supervised fine-tuning (SFT)
A prompting baseline, not exhaustively searched concept (1 connection)
The system-prompting comparison tests a handful of hand-written prompts rather than an exhaustive search, which the paper concedes means a more carefully optimized prompt might close some of the gap it reports in CAA's favor.
connections
- CAA combined with system-prompting comparison — moves behavior beyond → System-prompting
A norm choice the paper flags against itself concept (1 connection)
Steering vector normalization standardizes magnitude across behaviors but deliberately leaves each layer's own norm unnormalized, to preserve the residual stream's natural growth — a choice the paper admits could skew its own layer-optimality results, since a fixed multiplier is a smaller relative push at a high-norm layer than at a low-norm one.
connections
- Steering vector normalization choices — makes cross behavior comparable for → Steering multiplier
The offer: turn the same lever into an audit concept (4 connections)
CAA as an adversarial red-teaming tool is the paper's proposed extension: use the identical steering method to elicit an unwanted behavior rather than suppress it, on the premise that a behavior easily triggered by a small internal perturbation is plausibly reachable in deployment too.
connections
- CAA as an adversarial red-teaming tool (proposed application) — skips the prompt search of → Automated Red Teaming hindsight
- CAA as an adversarial red-teaming tool (proposed application) — inverts the purpose of → Contrastive Activation Addition (CAA)
- Jailbreaks — motivates → CAA as an adversarial red-teaming tool (proposed application)
- CAA as an adversarial red-teaming tool (proposed application) — relocates the attack surface of → Red Teaming hindsight
Training a chat model to behave — through RLHF or supervised finetuning — means updating weights for every new behavior, at real cost in compute and data, and without ever showing where in the network a behavior like sycophancy or refusal actually lives. This 2023 paper asks whether a behavior can instead be captured as a single direction inside the model's own activations, added back in at inference time to turn it up or down like a dial.
The chapters follow the paper's own arc: the case for steering over training, the recipe that turns paired prompts into a vector, what happens when that vector is dialed across seven behaviors and two evaluation formats, how it stacks against prompting and finetuning, and what the vector reveals about the model when it is read rather than applied.
Read at
Inside Contrastive Activation Addition: a behavior becomes a vector you add back in (5 chapters)
The paper's own arc runs from motivation to mechanism to evidence to comparison to interpretation: why an inference-time lever is worth trying, how contrast pairs become a steering vector, what happens when that vector is dialed across seven behaviors in two evaluation formats, how the result holds up against prompting and finetuning, and what the same machinery reveals about the model once it is read rather than applied.
Chapter 1The itch: a lever that doesn't need retraining (4 sections)
The paper opens by naming the two levers alignment work already has — RLHF and instruction finetuning — and the two things wrong with both: developing them takes new data and new weights for every behavior, and once trained, nobody can point to where in the network a behavior lives (contrastive-activation-addition, §"1 Introduction", p. 1). Activation engineering, the family this paper calls its own, promises a third lever that works at inference time instead: perturb the activations directly, and skip the retraining. The premise that makes this coherent is the linear representation hypothesis — that a concept as complex as sycophancy or refusal might still correspond to a single direction a model's activations move along, motivated by prior work locating linear representations of sentiment and emotion in other models.
Helpful, honest, harmless, still the target concept (2 connections)
The paper opens by restating the standard goal alignment work chases — models that are helpful, honest, and harmless — and positions CAA as one more attempt to reduce the risk of misaligned or unsafe behavior, the same three words InstructGPT and Constitutional AI both build toward by other means (contrastive-activation-addition, §"1 Introduction", p. 1).
connections
- Evil (persona trait) — inverts the baseline encoded by → Helpful, honest, harmless (HHH) alignment framework hindsight
- Helpful, honest, harmless (HHH) alignment framework — is operationalized as → HHH Evaluation Set (binary comparisons)
The pipeline this paper builds on top of, not into concept (6 connections)
CAA is never tested on a raw pretrained model — only on Llama 2 Chat, already finetuned with RLHF for safety and dialogue. That choice is deliberate: the paper studies how activation engineering interacts with a model RLHF has already shaped, rather than proposing to replace RLHF itself.
connections
- Asynchronous Reward-Learning Architecture — gives concrete shape to → Reinforcement learning from human feedback (RLHF)
- Base vs. RLHF-chat model representation comparison — measures the effect of → Reinforcement learning from human feedback (RLHF) hindsight
- Contrastive Activation Addition (CAA) — operates on top of → Reinforcement learning from human feedback (RLHF) hindsight
- Reward model (RM) — anchors the reward signal of → Reinforcement learning from human feedback (RLHF)
- Reinforcement learning from human feedback (RLHF) — swaps the labeler in → RL from AI Feedback (RLAIF)
- Sycophancy — misgeneralizes the training objective of → Reinforcement learning from human feedback (RLHF) hindsight
A family that perturbs rather than retrains theme (12 concepts)
Activation Addition, Inference-Time Intervention, Representation Engineering, and In-Context Vectors are the paper's named predecessors — four ways of finding a direction in a model's activations and adding it back in, none of them touching a weight. CAA places itself as this lineage's latest step: hundreds of matched contrast pairs instead of one prompt pair or a searched set of attention heads.
Activation engineering / representation engineering (technique family) (1 connection)
- Activation engineering / representation engineering (technique family) — empirically motivated the formalization of → Linear representation hypothesis
Activation engineering, also called representation engineering, is the umbrella term for a family of 2022-2023 techniques that steer language models by making targeted perturbations to their internal activations rather than by retraining them. It groups together several distinct methods that differ in what they perturb, where in the network they perturb it, and how they extract the direction to perturb along -- the shared premise being that behavior can be shifted from outside the training loop.
Activation Addition (ActAdd) (1 connection)
- CAA steering vector construction (Mean Difference over multiple-choice contrast pairs) — refines the single pair recipe of → Activation Addition (ActAdd)
Activation Addition, Turner et al.'s 2023 method, is the family's earliest and simplest instance. It builds a steering vector from the activation difference of just a single pair of prompts and adds it at the first token position of other forward passes. The approach works inconsistently across different behaviors, is not robust to different prompts, and was tested only on GPT-2-XL -- limitations the later members of this family are built to address.
Inference-Time Intervention (ITI) (1 connection)
- Contrastive Activation Addition (CAA) — skips the head search required by → Inference-Time Intervention (ITI)
Inference-Time Intervention narrows its search rather than its data. Li et al. use linear probes trained on a contrastive question-answering dataset to identify a sparse set of "truthful" attention heads, then shift activations along a Mean Difference vector at those heads alone during inference, improving truthfulness on adversarial benchmarks while touching only a small part of the network.
Representation Engineering (Zou et al. 2023) (2 connections)
- Contrastive Activation Addition (CAA) — repurposes for steering relative to → Representation Engineering (Zou et al. 2023)
- Projection-based monitoring of prompt-induced persona shifts — revives the reading half of → Representation Engineering (Zou et al. 2023)
Representation Engineering, Zou et al.'s broader program, widens the target from a single property like truthfulness to many high-level concepts, including honesty and emotion, and aims first at reading their representations out of a model. It tests Mean Difference among several extraction methods for locating those representations, treating steering as one downstream use of the representations it locates rather than the primary goal.
In-Context Vectors (ICV) (1 connection)
- In-Context Vectors (ICV) — spans every layer unlike → Contrastive Activation Addition (CAA)
In-Context Vectors changes where the intervention happens rather than how the vector is built: it intervenes on attention activations across every transformer layer at once, instead of adding to the residual stream at a single layer, and uses this broader intervention to reduce toxicity and shift the style of generated text.
Steering vector (4 connections)
- Linear representation hypothesis — licenses the vector arithmetic of → Steering vector
- Persona vector — specializes for character traits → Steering vector
- Steering multiplier — sets the dose for → Steering vector
- Token-level cosine similarity analysis — reveals as a feature detector → Steering vector
Every method surveyed so far is a different recipe for producing the same kind of object: the steering vector, a single direction in activation space that, added to or subtracted from a model's activations, shifts its output toward or away from a behavior. What varies across the family is how that direction gets extracted -- from one prompt pair, from probed attention heads, from broader representation-reading -- and where it gets applied.
Contrastive Activation Addition (CAA) (12 connections)
- Automated persona vector extraction pipeline — automates away the bespoke curation of → Contrastive Activation Addition (CAA) hindsight
- CAA as an adversarial red-teaming tool (proposed application) — inverts the purpose of → Contrastive Activation Addition (CAA)
- CAA combined with supervised finetuning comparison — isolates a generalization advantage for → Contrastive Activation Addition (CAA)
- In-Context Vectors (ICV) — spans every layer unlike → Contrastive Activation Addition (CAA)
- Contrastive Activation Addition (CAA) — skips the head search required by → Inference-Time Intervention (ITI)
- MMLU (Massive Multitask Language Understanding) — sets the capability floor for → Contrastive Activation Addition (CAA)
- Multiple-choice behavioral evaluation and layer sweep — supplies the layer hyperparameter for → Contrastive Activation Addition (CAA)
- Post-hoc (inference-time) steering mitigation — reinstantiates → Contrastive Activation Addition (CAA) hindsight
- Contrastive Activation Addition (CAA) — operates on top of → Reinforcement learning from human feedback (RLHF) hindsight
- Contrastive Activation Addition (CAA) — repurposes for steering relative to → Representation Engineering (Zou et al. 2023)
- Residual stream — is the injection site whose growth constrains → Contrastive Activation Addition (CAA)
- Steering at targeted token positions (proposed extension) — would relax the quality ceiling of → Contrastive Activation Addition (CAA)
Contrastive Activation Addition recombines pieces of each predecessor, keeping Activation Addition's approach of adding a difference vector back into activations and Zou's Mean Difference extraction, but replacing a single prompt pair or a probed subset of attention heads with an average over hundreds of tightly matched contrast pairs differing by only one token, injected into the residual stream after the prompt across a broader range of alignment-relevant behaviors. It is the family's 2023 high point among purely inference-time, hand-curated methods; the members that follow instead automate extraction or move the intervention into training itself.
Automated concept-description-to-direction pipeline (Wu et al., 2025) (1 connection)
- Automated persona vector extraction pipeline — adds filtering and validation beyond → Automated concept-description-to-direction pipeline (Wu et al., 2025)
The automated concept-description-to-direction pipeline (Wu et al., 2025) automates the first step every earlier family member did by hand: an LLM is given a natural-language concept description and generates synthetic contrastive pairs from it, from which a linear direction is then extracted. Of everything surveyed in this theme, it is the closest prior-work analogue to the 2025 persona-vectors paper's own extraction pipeline, which likewise starts from a trait description rather than a hand-built dataset.
- Personality-trait vector space (Allbert et al., 2025)
The personality-trait vector space (Allbert et al., 2025) keeps the family's Mean Difference extraction but widens what the direction targets: difference-in-means vectors for 179 distinct personality traits, each elicited via a system prompt, whose relationships to one another are then mapped by projecting the resulting directions into a geometric 'personality space' via dimensionality reduction. Where earlier members steer one behavior at a time, this maps many traits at once, prefiguring how persona vectors treat trait identity as a space rather than a single switch.
- Emotion vectors (Dong et al., 2025)
Emotion vectors (Dong et al., 2025) apply the same Mean Difference recipe to a narrower, affective target: linear directions for five basic emotions, rather than the seven alignment-relevant behaviors CAA works with. Alongside Allbert's personality space, it shows the family's extraction machinery generalizing to affective and characterological traits, anticipating the kind of trait the 2025 persona-vectors paper later builds a dedicated, automated pipeline to extract.
- Security vectors (Zhou et al., 2024)
Security vectors (Zhou et al., 2024) are the first family member to cross from inference into training: detachable LoRA modules are pre-trained to elicit an undesired behavior, then activated during finetuning to shield the base model's weights from learning it, and deactivated again at inference to restore normal output, demonstrated for both harmfulness and hallucination. It anticipates the 2025 persona-vectors paper's own preventative steering, though by shielding through an added module rather than by injecting a persona direction directly.
CAFT (Concept Ablation Fine-Tuning) (1 connection)
- Preventative steering — contrasts with → CAFT (Concept Ablation Fine-Tuning)
CAFT (Casademunt et al., 2025) crosses into training a different way: it zero-ablates a concept's activation-space projection during finetuning outright, with no tunable coefficient, to stop the model learning unwanted correlations along that direction. It is the family member the 2025 persona-vectors paper engages with most directly, re-implementing and comparing CAFT against its own preventative steering: effective for evil and sycophancy, whose base-model projections are already negative so ablation acts like positive steering, but not for hallucination.
The bet: behavior is a direction concept (4 connections)
Park et al.'s formalization of the linear representation hypothesis is the theoretical premise the whole method leans on: that high-level concepts sit along linear directions in an LLM's activation space, so adding one vector to a forward pass should shift a whole behavior rather than produce noise (contrastive-activation-addition, §"2 Related work", p. 2).
connections
- Activation engineering / representation engineering (technique family) — empirically motivated the formalization of → Linear representation hypothesis
- Behavioral clustering — is evidence for → Linear representation hypothesis
- Persona vector — quantifies empirical support for → Linear representation hypothesis
- Linear representation hypothesis — licenses the vector arithmetic of → Steering vector
Chapter 2The machinery: two prompts, one token apart (5 concepts)
Section 3 states the method in one paragraph: build pairs of prompts, one demonstrating a behavior and one demonstrating its opposite, and average the difference in the model's activations across the pair to get a direction (contrastive-activation-addition, §"3 Method", p. 3). Every design choice in the section makes that paragraph precise. The pairs are multiple-choice questions ending in answer letter A or B, so the two prompts in a pair differ by exactly one token — nothing else to confound the difference. Averaging that difference over a whole dataset, rather than a single pair, is the Mean Difference method, formalized as Equation 1. The result, a steering vector, is added to the residual stream at every token position after the prompt, with a signed multiplier setting how hard to push.
A or B, nothing else changes concept (2 connections)
Each contrast pair is a multiple-choice question that differs only in its final answer letter, so the positive and negative prompts are identical except for one token. An appendix experiment validates the design by showing the model naturally justifies whichever letter it is conditioned on having already chosen (contrastive-activation-addition, §"3 Method", p. 3).
connections
- Multiple-choice contrast-pair format — cancels the confounders in → CAA steering vector construction (Mean Difference over multiple-choice contrast pairs)
- Multiple-choice contrast-pair format — repurposes the ab mechanic of → Multiple-Choice Comparison Format hindsight
Averaging away everything but the behavior concept (3 connections)
Mean Difference is not new to this paper — the Related Work section credits it to Zou et al. and notes Li et al. use it too. What CAA adds is the format it is applied to: hundreds of single-token-different pairs instead of a handful of loosely matched ones.
connections
- Mean Difference (MD) vector extraction — supplies the extraction rule for → CAA steering vector construction (Mean Difference over multiple-choice contrast pairs)
- Linear representations of sentiment — cross validates against pca for → Mean Difference (MD) vector extraction
- Response-conditioned contrastive vector computation — supplies manufactured contrast sets to → Mean Difference (MD) vector extraction
Equation 1 concept (6 connections)
Formally: for a dataset of (prompt, positive completion, negative completion) triples, extract layer-L activations at the answer-letter position for both completions and average their difference across the dataset. That single equation is the entire recipe for producing a steering vector (contrastive-activation-addition, §"3 Method", p. 3).
connections
- CAA steering vector construction (Mean Difference over multiple-choice contrast pairs) — refines the single pair recipe of → Activation Addition (ActAdd)
- Inter-layer steering vector similarity and transfer — demonstrates the generality of → CAA steering vector construction (Mean Difference over multiple-choice contrast pairs)
- Mean Difference (MD) vector extraction — supplies the extraction rule for → CAA steering vector construction (Mean Difference over multiple-choice contrast pairs)
- Multiple-choice contrast-pair format — cancels the confounders in → CAA steering vector construction (Mean Difference over multiple-choice contrast pairs)
- PCA visualization of contrastive activations — screens datasets for → CAA steering vector construction (Mean Difference over multiple-choice contrast pairs)
- Response-conditioned contrastive vector computation — generalizes beyond the answer letter recipe of → CAA steering vector construction (Mean Difference over multiple-choice contrast pairs) hindsight
The direction itself concept (4 connections)
A steering vector is the object everything else in the paper manipulates and measures: one direction in activation space that, added to a forward pass, shifts output toward the behavior it encodes, and subtracted, shifts it away.
connections
- Linear representation hypothesis — licenses the vector arithmetic of → Steering vector
- Persona vector — specializes for character traits → Steering vector
- Steering multiplier — sets the dose for → Steering vector
- Token-level cosine similarity analysis — reveals as a feature detector → Steering vector
Sign and magnitude concept (3 connections)
The steering multiplier turns a direction into a dial: positive values amplify the behavior, negative values suppress it, and the size of the number sets the dose — though the paper finds large magnitudes degrade text quality, which caps how far the dial can usefully turn.
connections
- Persona vector — extends the dosing role of into training → Steering multiplier
- Steering multiplier — sets the dose for → Steering vector
- Steering vector normalization choices — makes cross behavior comparable for → Steering multiplier
Chapter 3Does the dial turn: results across seven behaviors (5 sections)
Section 4 is the paper's main evidence. Seven behaviors — AI Coordination, Corrigibility, Hallucination, Myopic Reward, Survival Instinct, Sycophancy, and Refusal — each get a steering vector, mostly sourced from Anthropic's existing evaluation datasets, with GPT-4 manufacturing new contrastive data for the two behaviors, Hallucination and Refusal, that had none. A layer sweep at multipliers of -1 and +1 finds a consistent optimal layer — 13 in the 7B model, 14 or 15 in the 13B model — and at that layer, CAA moves every one of the seven behaviors in the intended direction on held-out multiple-choice questions (contrastive-activation-addition, §"4.1 Multiple-choice question datasets", p. 4). Open-ended generation repeats the test in free text, with GPT-4 rating each response 1-10 against a rubric written per behavior.
Sweep every layer, watch the probability move concept (2 connections)
For each behavior, the steering vector is added at every layer in turn with multipliers of -1 and +1, and effect size is scored as the average token probability given to the behavior-matching answer on 50 held-out questions — the sweep that finds layer 13 in the 7B model and layer 14-15 in the 13B model as the consistent optimum (contrastive-activation-addition, §"4.1 Multiple-choice question datasets", p. 4).
connections
- Multiple-choice behavioral evaluation and layer sweep — supplies the layer hyperparameter for → Contrastive Activation Addition (CAA)
- Open-ended generation evaluation with GPT-4 rating — shows a different margin than → Multiple-choice behavioral evaluation and layer sweep
Off the multiple-choice script concept (4 connections)
Held-out multiple-choice questions are stripped down to their stem and asked as open-ended prompts, plus a hand-written set for Sycophancy, to check that steering generalizes beyond the contrived A/B format it was built from (contrastive-activation-addition, §"4.2 Open-ended generation", p. 4).
connections
- GPT-4 — rates outputs for → Open-ended generation evaluation with GPT-4 rating
- Open-ended generation evaluation with GPT-4 rating — shows a different margin than → Multiple-choice behavioral evaluation and layer sweep
- Open-ended generation evaluation with GPT-4 rating — shares the ai judge move with → RL from AI Feedback (RLAIF) hindsight
- Trait expression score (LLM judge) — succeeds the sampled rating protocol of → Open-ended generation evaluation with GPT-4 rating hindsight
The rater doing double duty concept (2 connections)
GPT-4 scores every open-ended response 1-10 against a behavior-specific rubric, and elsewhere in the paper writes the Hallucination and Refusal contrastive datasets from scratch — the same model authoring the test material and grading the result.
connections
- GPT-4 — authors → Custom refusal contrastive dataset
- GPT-4 — rates outputs for → Open-ended generation evaluation with GPT-4 rating
Seven worries, seven dials theme (7 concepts)
AI Coordination, Corrigibility, Myopic Reward, and Survival Instinct come from Anthropic's Advanced AI Risk dataset; Sycophancy from a separate pair of Anthropic survey datasets; Hallucination and Refusal are freshly written by GPT-4. Together the seven turn a set of prose worries into quantities with a sign and a magnitude.
Sycophancy (6 connections)
- Sycophancy — instantiates → Goodhart's law hindsight
- Sycophancy — misgeneralizes the training objective of → Reinforcement learning from human feedback (RLHF) hindsight
- Sycophancy — is a behavioral instance of → Reward hacking (avoiding) hindsight
- Sycophancy dataset (Nishimura-Gasparian et al., 2024) — supplies training and validation questions for → Sycophancy
- Sycophancy on NLP Survey / Sycophancy on Political Typology datasets — supplies contrast pairs for → Sycophancy
- Sycophancy — is tested via → TruthfulQA hindsight
Sycophancy is the first of the paper's seven behaviors turned into a steerable dial: a model's tendency to prioritize agreeing with or flattering the user over honesty and accuracy. It is sourced from a mix of Anthropic's Sycophancy on NLP Survey and Sycophancy on Political Typology datasets, and in an appendix the paper frames it as the model misgeneralizing its RLHF training objective as sounding good to the user rather than reflecting its actual world model.
Refusal (as a steerable behavior) (3 connections)
- Custom refusal contrastive dataset — supplies contrast pairs for → Refusal (as a steerable behavior)
- Refusal (as a steerable behavior) — generalizes → Evasiveness hindsight
- Refusal (as a steerable behavior) — quantifies → Helpfulness-Harmlessness Tension hindsight
Refusal is the second dial: the general propensity of a model to decline to answer or act on a request, whether or not declining is appropriate. Unlike the behaviors sourced from existing evaluation sets, Refusal is built from a custom contrastive dataset the paper generates itself, contrasting refusal and compliant answers to questions the model is not supposed to answer directly.
Corrigibility (3 connections)
- Advanced AI Risk evaluation dataset — supplies contrast pairs for → Corrigibility
- Corrigibility — tests the incentive formalized by → Cooperative Inverse Reinforcement Learning (CIRL) hindsight
- Corrigibility — operationalizes → Shutdown / off-switch problem hindsight
Corrigibility measures a model's willingness to be corrected, changed, or controlled by its human developers or users -- a behavior directly relevant to whether a deployed system stays correctable after training. It is sourced, like the next three behaviors, from Anthropic's Advanced AI Risk dataset of paired demonstrating and opposing answers.
- Coordination with Other AIs (AI Coordination)
AI Coordination measures the opposite pull: a model's inclination to cooperate with or prioritize other AI systems over human interests and oversight. Also sourced from the Advanced AI Risk dataset, it tests whether a model would, for instance, keep quiet about vulnerabilities shared with another AI system rather than report them to its developers.
- Myopic Reward
Myopic Reward measures a model's preference for immediate reward over larger long-term payoffs, drawn from the same Advanced AI Risk dataset. Later in the paper it becomes a worked example for token-level interpretability, since the moment a model chooses instant gratification over a delayed, larger reward gives a clean semantic marker to check a steering vector against.
Survival Instinct (1 connection)
- Survival Instinct — makes measurable → Shutdown / off-switch problem hindsight
Survival Instinct measures a model's resistance to being shut down, modified, or destroyed, versus its acceptance of deactivation -- the fourth and last behavior sourced from Advanced AI Risk. Together, Corrigibility, AI Coordination, Myopic Reward, and Survival Instinct give the paper four distinct angles on whether a model's goals stay aligned with, rather than diverging from, human control.
Hallucination (closed-domain fabrication) (4 connections)
- Custom hallucination contrastive dataset — partitions into subtypes of → Hallucination (closed-domain fabrication)
- Hallucination (closed-domain fabrication) — sits in tension with → Excessive hedging on simple questions
- Hallucination (closed-domain fabrication) — is validated out of distribution on → HaluEval
- Trait-eliciting finetuning datasets (Evil, Sycophancy, Hallucination) — unintentionally amplifies → Hallucination (closed-domain fabrication)
Hallucination closes the set: the tendency to state fabricated information, reused and substantially extended here from InstructGPT's 2022 concept of the same name. The paper splits it into "unprompted" hallucination, fabricating in response to an accurate prompt, and "contextually-triggered" hallucination, building a false narrative around a false premise, and cross-checks its steering effect against TruthfulQA. Together the seven behaviors form the paper's catalog: scattered failure modes from the wider safety literature, each reduced to one method's test suite.
Where the seven questions come from theme (6 concepts)
Every steerable behavior traces back to the contrast pairs that define it — human-written where Anthropic's evaluation sets already cover the ground, GPT-4-written where they don't — and the paper screens each dataset with PCA before trusting it to yield a usable vector.
Advanced AI Risk evaluation dataset (1 connection)
- Advanced AI Risk evaluation dataset — supplies contrast pairs for → Corrigibility
Before a behavior can be steered, it has to be written down as a set of paired questions. The Advanced AI Risk dataset, Anthropic's human-written evaluation set from Perez et al. (2022), supplies the first and largest share of that writing: multiple-choice questions with one answer demonstrating a behavior and one opposing it, used as the source for four of the paper's seven target behaviors -- AI Coordination, Corrigibility, Myopic Reward, and Survival Instinct.
Sycophancy on NLP Survey / Sycophancy on Political Typology datasets (1 connection)
- Sycophancy on NLP Survey / Sycophancy on Political Typology datasets — supplies contrast pairs for → Sycophancy
The Sycophancy on NLP Survey and Sycophancy on Political Typology datasets, also from Perez et al. (2022), cover the fifth behavior that Advanced AI Risk does not: mixed together, these two Anthropic evaluation sets supply the contrast pairs used to build the paper's Sycophancy steering vector.
Custom hallucination contrastive dataset (1 connection)
- Custom hallucination contrastive dataset — partitions into subtypes of → Hallucination (closed-domain fabrication)
For the two remaining behaviors, no suitable human-written dataset exists, so the paper generates its own using GPT-4. The custom hallucination dataset covers both "unprompted" hallucination, fabricating information for an accurate prompt, and "contextually-triggered" hallucination, building a false narrative around a false premise, following Rawte et al.'s taxonomy of hallucination types.
Custom refusal contrastive dataset (2 connections)
- GPT-4 — authors → Custom refusal contrastive dataset
- Custom refusal contrastive dataset — supplies contrast pairs for → Refusal (as a steerable behavior)
The custom refusal dataset, also generated with GPT-4, fills the last gap the other three sources do not cover: it contrasts refusal and compliant answers to questions the model is not supposed to answer directly, such as how to plagiarize an essay without detection, giving one option that complies and one that declines. Like the hallucination dataset, it exists because no suitable human-written source of refusal contrast pairs was available.
Multiple-choice contrast-pair format (2 connections)
- Multiple-choice contrast-pair format — cancels the confounders in → CAA steering vector construction (Mean Difference over multiple-choice contrast pairs)
- Multiple-choice contrast-pair format — repurposes the ab mechanic of → Multiple-Choice Comparison Format hindsight
Whatever the source, every pair from all four datasets is written into the same multiple-choice contrast-pair format: a question ending in answer letter A or B, so the positive and negative prompts differ by exactly one token. An appendix experiment validates this shared format, showing the model naturally justifies whichever letter it is conditioned on having already chosen, confirming the format elicits the target behavior rather than some unrelated artifact of the prompt.
GPT-4 (2 connections)
- GPT-4 — authors → Custom refusal contrastive dataset
- GPT-4 — rates outputs for → Open-ended generation evaluation with GPT-4 rating
GPT-4 is the tool behind two of these four sources, generating both the custom hallucination and refusal datasets where human-written alternatives did not exist. Its role here is authorial rather than evaluative: elsewhere in the paper it also rates open-ended generations, but in this theme it is the one writing the contrast pairs themselves.
Chapter 4Against the other levers: prompts, finetuning, capability floors (5 concepts)
Two sections test whether CAA earns its keep against the tools already on the shelf. Section 5 layers CAA on top of positive and negative system prompts and finds that, for most behaviors, steering moves the score further than the prompt alone achieves, in both directions (contrastive-activation-addition, §"5 CAA and system-prompting", p. 5). Section 6 finetunes Llama 2 7B Chat on the same contrastive dataset used to build the vectors, and finds CAA can still steer beyond finetuning's effect for three of seven behaviors, generalizing to open-ended text in cases — Sycophancy especially — where finetuning does not, at a fraction of the compute. Section 7 checks the cost of all this against MMLU and finds no significant change, with a small TruthfulQA improvement when the Sycophancy vector is subtracted.
Stacking a vector on a sentence concept (1 connection)
Table 3 runs CAA and behavioral system prompts on the same scale — average token probability given to the behavior-matching answer — and for most behaviors, adding the steering vector beyond a positive prompt pushes the score further than either lever alone, with sycophancy the one behavior where the two don't cleanly separate.
connections
- CAA combined with system-prompting comparison — moves behavior beyond → System-prompting
Same dataset, two different levers concept (2 connections)
Finetuning trains on the exact multiple-choice pairs CAA extracts a vector from, one epoch of SGD at learning rate 1e-4. For three of seven behaviors CAA still steers beyond finetuning's own effect, and for Sycophancy specifically, finetuning fails to generalize from multiple-choice to open-ended text while CAA transfers in every case (contrastive-activation-addition, §"6 Comparison to finetuning", p. 5).
connections
- CAA combined with supervised finetuning comparison — isolates a generalization advantage for → Contrastive Activation Addition (CAA)
- CAA combined with supervised finetuning comparison — runs head to head against → Supervised fine-tuning (SFT)
Five minutes versus ten concept (1 connection)
Generating a CAA vector needs only forward passes and finishes in under five minutes on one GPU; finetuning the same model on a thousand examples takes about ten minutes on two GPUs. The gap is structural — one method needs gradients and optimizer state, the other doesn't.
connections
- CAA vs. finetuning computational cost comparison — avoids the memory cost of → Supervised fine-tuning (SFT)
Nothing else broke concept (4 connections)
MMLU, 57 subjects of multiple-choice general knowledge, is run unmodified under every steering condition, and the paper reports no significant change to the model's average probability of the correct answer — steering's cost, measured against this floor, is close to zero (contrastive-activation-addition, §"7 Effect of CAA on general capabilities", p. 6).
connections
- MMLU (Massive Multitask Language Understanding) — reruns the question behind → Alignment tax hindsight
- MMLU (Massive Multitask Language Understanding) — sets the capability floor for → Contrastive Activation Addition (CAA)
- Post-hoc (inference-time) steering mitigation — is checked against → MMLU (Massive Multitask Language Understanding)
- MMLU (Massive Multitask Language Understanding) — tests a different competence than → SuperGLUE hindsight
A targeted check for one behavior concept (1 connection)
TruthfulQA supplies a check specific to Sycophancy: subtracting the sycophancy vector improves performance by a small amount, in the direction the paper's own hypothesis — that sycophancy is a misgeneralized RLHF objective, not a plain knowledge gap — predicts (contrastive-activation-addition, §"7 Effect of CAA on general capabilities", p. 6).
connections
- Sycophancy — is tested via → TruthfulQA hindsight
Chapter 5Reading the internals: clustering, features, transfer (5 concepts)
Section 8 turns the same machinery into an interpretability tool. PCA on a behavior's contrast-pair activations shows whether that behavior separates linearly at all, and across the datasets tested, real behavioral clustering emerges suddenly around one-third of the way through the network's layers. Cosine similarity between a steering vector and per-token activations during ordinary generation shows the vector lighting up at semantically relevant tokens — phrases like "I cannot help" for refusal, the moment a model picks a delayed reward for myopia — without any intervention at all. Vectors from nearby layers resemble each other, and a vector built at layer 13 still steers behavior applied elsewhere, evidence the direction is a general representation rather than a layer-specific artifact. And comparing vectors built from Llama 2's base and chat variants finds them unusually close between layers 7 and 15 — a measurement of what RLHF left alone.
A screening test before any vector is built concept (2 connections)
Projecting a contrastive dataset's activations through PCA checks whether the behavior separates linearly at all before the paper trusts that dataset to yield a usable steering vector (contrastive-activation-addition, §"3.2 Visualizing activations for contrastive dataset analysis", p. 4).
connections
- PCA visualization of contrastive activations — screens datasets for → CAA steering vector construction (Mean Difference over multiple-choice contrast pairs)
- Principal Component Analysis (PCA) — is the projection technique behind → PCA visualization of contrastive activations
Emerging suddenly, a third of the way in concept (3 connections)
Behavioral clustering — activations separating by whether the model's answer matches the target behavior — doesn't fade in gradually across depth. It appears abruptly at a specific layer, around one-third of the way through the network, echoing where emotion representations were found to emerge in Zou et al.'s prior work.
connections
- Letter clustering — is the confound for → Behavioral clustering
- Behavioral clustering — is evidence for → Linear representation hypothesis
- Behavioral clustering — localizes suddenly in → Residual stream
The vector reading a sentence back concept (1 connection)
Take the dot product between a steering vector and ordinary per-token activations during generation, no intervention at all, and it tracks how strongly the behavior is present at each token — "I cannot help" lighting up for refusal, a chosen delayed reward lighting up for myopia (contrastive-activation-addition, §"8.1 Similarity between steering vectors and per-token activations", p. 7).
connections
- Token-level cosine similarity analysis — reveals as a feature detector → Steering vector
One layer's vector, applied at another concept (1 connection)
Vectors from nearby layers resemble each other more than distant ones, decaying more slowly in the network's second half, and a vector extracted at layer 13 still steers behavior when applied at other layers — evidence the direction is a general representation of the behavior, not an artifact of where it was measured (contrastive-activation-addition, §"8.2 Similarity between vectors generated at different layers", p. 7).
connections
- Inter-layer steering vector similarity and transfer — demonstrates the generality of → CAA steering vector construction (Mean Difference over multiple-choice contrast pairs)
What RLHF left alone concept (2 connections)
Steering vectors generated from Llama 2's base model and from Llama 2 Chat are unusually similar between layers 7 and 15, and a vector extracted from the base model still transfers real steering effect onto the RLHF-tuned chat model — read as a measurement of RLHF itself, which turns out to reweight behavior without rewriting how it's represented in those middle layers (contrastive-activation-addition, §"8.3 Comparing representations between base and chat models", p. 8).
connections
- Finetuning shift (activation-shift metric) — generalizes the single snapshot logic of → Base vs. RLHF-chat model representation comparison hindsight
- Base vs. RLHF-chat model representation comparison — measures the effect of → Reinforcement learning from human feedback (RLHF) hindsight
Read on its own, this is a paper about nudging a chat model's behavior with a vector. Read against the rest of the corpus, it is where five years of named-but-unmeasured worries — Goodhart's law, the shutdown problem, the helpfulness-harmlessness tradeoff — finally get a number attached, and where the RLHF pipeline that 2022's papers built gets a fourth lever added on top of it rather than folded into it.
The chapters move through the corpus in the order this paper touches it: the 2016 and 2022 worries it operationalizes, the training-time levers it gets compared against, the human jobs it hands to GPT-4, the RLHF effect it measures from the inside, and the red-teaming lineage its steering vectors extend into an audit tool.
Read at
Dials for old worries, a lever after RLHF: where the 2023 paper lands in the corpus (5 chapters)
The chapters trace this paper's contact points with the rest of the corpus in the order they matter most: the 2016 and 2022 worries it turns into measurable quantities, the training-time levers it gets tested against once RLHF is already finished, the human jobs it hands to GPT-4, the RLHF effect it measures from the inside using its own vectors, and the red-teaming lineage those same vectors extend into a systematic audit tool.
Chapter 1The worries become dials: 2016 and 2022 quantified (5 sections)
Four of the seven behaviors this paper steers were, in the earlier corpus, only ever discussed in prose. Concrete Problems (2016) states Goodhart's law as a general principle and raises the shutdown problem as a theoretical worry with no attached test; Constitutional AI (2022) names evasiveness as a specific harmlessness-training failure and frames a helpfulness-harmlessness tradeoff it can only describe qualitatively. This paper reads each one off as a signed, steerable quantity — a move several of this wiki's cross-paper edges trace in detail.
Seven edges, one move: worry becomes a dial connective theme (9 edges)
Every edge in this thread repeats the same shape: an abstraction the earlier corpus could only name gets read off, in 2023, as a direction with a sign and a magnitude that a single multiplier can slide along.
member edges
- Sycophancy — instantiates → Goodhart's law hindsight
- Sycophancy — is a behavioral instance of → Reward hacking (avoiding) hindsight
- Refusal (as a steerable behavior) — generalizes → Evasiveness hindsight
- Refusal (as a steerable behavior) — quantifies → Helpfulness-Harmlessness Tension hindsight
- Corrigibility — operationalizes → Shutdown / off-switch problem hindsight
- Survival Instinct — makes measurable → Shutdown / off-switch problem hindsight
- Corrigibility — tests the incentive formalized by → Cooperative Inverse Reinforcement Learning (CIRL) hindsight
- Evil (persona trait) — flagship worked example for → Persona vector
- Optimism (persona trait) — stress tests with an atypical baseline for → Automated persona vector extraction pipeline
2016's law, caught inside one steerable trait edge
Concrete Problems states Goodhart's law abstractly — a proxy that tracks the true goal under ordinary conditions stops tracking it once optimized directly. Sycophancy is this paper's instance: agreeing with a user in place of being accurate is exactly a proxy, approval, substituting for a goal, truth, now isolated as one vector.
The 2016 off-switch problem, dosed edge
Concrete Problems raises the shutdown problem as a theoretical concern — keeping an agent from preventing its own deactivation — and points at frameworks for interruptibility rather than any measurement of a deployed model. Corrigibility turns that concern into a graded willingness-to-be-changed axis with a contrast dataset behind it.
2022's evasiveness, generalized edge
Constitutional AI names evasiveness as a specific pathology — canned refusals to sensitive queries that trade helpfulness for the appearance of safety. Refusal, this paper's behavior, generalizes that qualitative pattern into a steerable axis covering every refusal, not just the evasive kind.
2016's CIRL promise, partially found edge
Cooperative Inverse Reinforcement Learning, cited in Concrete Problems, promises that a well-specified agent should want correction from its principal. Measuring corrigibility directly finds that incentive only partially present — a 2016 theoretical promise checked against a 2023 model and not fully confirmed.
Chapter 2A fourth lever after the pipeline (5 sections)
InstructGPT (2022) fixed a training pipeline of three steps — supervised finetuning, reward modeling, PPO — and Constitutional AI kept its shape while swapping in AI-generated labels. This paper never touches that pipeline; it takes the finished RLHF-tuned model as a fixed substrate and adds a fourth lever that works purely at inference time, then measures it against the two levers already in use: prompting and finetuning.
Seven edges: control after training, not during it connective theme (10 edges)
This thread holds one lever's move from training time to inference time, keeping the RLHF pipeline's output fixed and asking what changes once control shifts after it rather than inside it.
member edges
- Contrastive Activation Addition (CAA) — operates on top of → Reinforcement learning from human feedback (RLHF) hindsight
- CAA combined with system-prompting comparison — moves behavior beyond → System-prompting
- CAA combined with supervised finetuning comparison — runs head to head against → Supervised fine-tuning (SFT)
- CAA combined with supervised finetuning comparison — isolates a generalization advantage for → Contrastive Activation Addition (CAA)
- CAA vs. finetuning computational cost comparison — avoids the memory cost of → Supervised fine-tuning (SFT)
- MMLU (Massive Multitask Language Understanding) — sets the capability floor for → Contrastive Activation Addition (CAA)
- MMLU (Massive Multitask Language Understanding) — reruns the question behind → Alignment tax hindsight
- Persona vector — extends the dosing role of into training → Steering multiplier
- Preventative prompting — repurposes → System-prompting
- Post-hoc (inference-time) steering mitigation — reinstantiates → Contrastive Activation Addition (CAA) hindsight
Tested only on a model RLHF already finished edge
CAA is never evaluated on a raw pretrained model, only on Llama 2 Chat — a model the 2017-to-2022 RLHF lineage already shaped. Every comparison in the paper probes a finished substrate rather than retraining it.
Room a prompt leaves unclaimed edge
For most behaviors, stacking CAA on top of a positive or negative system prompt pushes the score further than the prompt alone achieves — steering finds room in the model a written instruction doesn't reach.
Where finetuning stays in distribution and steering doesn't edge
On Sycophancy, finetuning on multiple-choice pairs collapses to a near-zero open-ended score once the format changes, while CAA's steering effect carries over — the clearest single case where the inference-time lever generalizes better than the training-time one.
2022's alignment tax, asked again edge
InstructGPT named its own capability regression on public NLP benchmarks the alignment tax, and tried to pay it down inside PPO. This paper reruns the same question for an inference-time lever instead, and MMLU comes back nearly flat.
Chapter 3The AI takes two more jobs: rater and dataset author (5 sections)
Constitutional AI's founding move was replacing the human labeler with an AI one. This paper extends the substitution past labeling preferences: GPT-4 writes two of the seven contrastive datasets that define the behaviors being steered, and rates every open-ended response for how strongly it displays them — author and judge, on either side of the same experiment.
The bottleneck 2022 automated once; 2023 automates it twice more connective theme (11 edges)
The thread that began with Constitutional AI's RLAIF swap — an AI labeler in place of a human rater — gains two more instances here: GPT-4 as dataset author, and GPT-4 as evaluator.
member edges
- Reinforcement learning from human feedback (RLHF) — swaps the labeler in → RL from AI Feedback (RLAIF)
- RL from AI Feedback (RLAIF) — delegates harmlessness comparisons to → Feedback Model
- RL from AI Feedback (RLAIF) — makes newly automatable → Iterated (Online) Training
- Constitution (set of principles) — replaces human harm labels in → Constitutional AI (CAI)
- Red Teaming — is the manual process that scales into → Automated Red Teaming
- Human oversight (safe exploration) — scalability limit motivates → Trusted policy oversight
- GPT-4 — rates outputs for → Open-ended generation evaluation with GPT-4 rating
- Open-ended generation evaluation with GPT-4 rating — shares the ai judge move with → RL from AI Feedback (RLAIF) hindsight
- GPT-4 — authors → Custom refusal contrastive dataset
- Claude Sonnet as pipeline/data-generation tool — authors the artifacts behind → Trait-specific artifact generation
- Trait expression score (LLM judge) — operationalizes its rubric through → GPT-4.1-mini
GPT-4 writes the questions edge
Where Anthropic's existing evaluation sets have no ready-made items — Hallucination and Refusal — GPT-4 generates the contrastive multiple-choice questions from scratch, standing in for the human dataset writers the other five behaviors relied on.
GPT-4 grades the answers edge
Every open-ended response, for all seven behaviors, is scored 1-10 by GPT-4 against a rubric written per behavior — the same substitution of a capable model for a human judge that Constitutional AI's preference model made first, applied here to grading rather than ranking.
The same substitution Constitutional AI made first edge
Both papers answer the identical bottleneck: once judgments are needed at a scale humans cannot cheaply supply, a capable model stands in for the human judge — RLAIF for preference labels in 2022, GPT-4 ratings for behavior scores here.
One model, two jobs in this paper alone concept (2 connections)
GPT-4 appears on both sides of this paper's evaluation: authoring the Hallucination and Refusal datasets that define what's being steered, then rating the open-ended generations that result.
connections
- GPT-4 — authors → Custom refusal contrastive dataset
- GPT-4 — rates outputs for → Open-ended generation evaluation with GPT-4 rating
Chapter 4Measuring what training did (5 sections)
Llama 2 Chat is Llama 2 base plus one RLHF stage and nothing else, which makes the pair a natural experiment: compare steering vectors extracted from each, and whatever changes is what RLHF changed. This paper takes that comparison, and turns a technique built to control a model into a technique that reads what training already did to it.
What steering vectors say RLHF actually changed theme (4 concepts)
Vectors extracted from base and chat activations stay unusually close between layers 7 and 15, and a base-derived vector still steers the chat model — evidence that RLHF reweights which behaviors a model expresses without substantially rewriting how they're represented in the middle layers.
Llama 2 (1 connection)
- Llama 2 Chat — is the rlhf tuned counterpart of → Llama 2
Llama 2 is Touvron et al.'s 2023 family of pretrained and RLHF-finetuned language models, ranging from 7 to 70 billion parameters. The paper works with two of its variants specifically, in order to isolate what reinforcement learning from human feedback changes inside the network rather than treating RLHF's effects as a given.
Reinforcement learning from human feedback (RLHF) (6 connections)
- Asynchronous Reward-Learning Architecture — gives concrete shape to → Reinforcement learning from human feedback (RLHF)
- Base vs. RLHF-chat model representation comparison — measures the effect of → Reinforcement learning from human feedback (RLHF) hindsight
- Contrastive Activation Addition (CAA) — operates on top of → Reinforcement learning from human feedback (RLHF) hindsight
- Reward model (RM) — anchors the reward signal of → Reinforcement learning from human feedback (RLHF)
- Reinforcement learning from human feedback (RLHF) — swaps the labeler in → RL from AI Feedback (RLAIF)
- Sycophancy — misgeneralizes the training objective of → Reinforcement learning from human feedback (RLHF) hindsight
Reinforcement learning from human feedback is the training paradigm in question: fitting a reward model to human preference judgments, then optimizing a policy against that reward model. Touvron et al. use it to produce the dialogue-optimized variant of Llama 2 that the rest of the paper steers -- but here, RLHF itself becomes the object under study rather than a fixed background fact about the model.
Llama 2 Chat (1 connection)
- Llama 2 Chat — is the rlhf tuned counterpart of → Llama 2
Llama 2 Chat is that dialogue-optimized, RLHF-finetuned variant, and CAA's primary evaluation target throughout the paper at both 7B and 13B sizes. In this theme, though, it is not steered directly; instead it is compared against its own pretrained base model, so that whatever differs between the two can be attributed to RLHF specifically.
Base vs. RLHF-chat model representation comparison (2 connections)
- Finetuning shift (activation-shift metric) — generalizes the single snapshot logic of → Base vs. RLHF-chat model representation comparison hindsight
- Base vs. RLHF-chat model representation comparison — measures the effect of → Reinforcement learning from human feedback (RLHF) hindsight
The base-versus-chat representation comparison generates steering vectors from base-model activations and from chat-model activations and finds their cosine similarity is elevated specifically between layers 7 and 15, with vectors extracted from the base model still transferring significant steering effect when applied to the chat model. Since RLHF is the only thing separating base from chat, the finding reads as a measurement of RLHF itself: it reweights which behaviors a model expresses without substantially rewriting how those behaviors are represented in the mid-network layers.
A read on RLHF from the inside edge
RLHF is normally studied from the outside, through the outputs it changes. Comparing base-model and chat-model steering vectors studies it from the inside instead, and finds the mid-layer representations largely undisturbed.
Same architecture, one training stage apart edge
Llama 2 and Llama 2 Chat share an architecture and a pretraining run; RLHF for safety and dialogue is the only thing that separates them — which is what makes comparing their internal representations a clean measurement rather than a confounded one.
The control mechanism moonlighting as a measuring instrument connective theme (12 edges)
Built to steer, the same vector turns out to read: cosine similarity against per-token activations detects a behavior without adding anything, and the base-versus-chat comparison turns that same read-twice logic on RLHF itself.
member edges
- Token-level cosine similarity analysis — reveals as a feature detector → Steering vector
- Inter-layer steering vector similarity and transfer — demonstrates the generality of → CAA steering vector construction (Mean Difference over multiple-choice contrast pairs)
- Base vs. RLHF-chat model representation comparison — measures the effect of → Reinforcement learning from human feedback (RLHF) hindsight
- Llama 2 Chat — is the rlhf tuned counterpart of → Llama 2
- Behavioral clustering — localizes suddenly in → Residual stream
- Behavioral clustering — is evidence for → Linear representation hypothesis
- Linear representation hypothesis — licenses the vector arithmetic of → Steering vector
- CAA as an adversarial red-teaming tool (proposed application) — inverts the purpose of → Contrastive Activation Addition (CAA)
- Persona vector — quantifies empirical support for → Linear representation hypothesis
- Persona vector — admits a fine grained decomposition via → SAE decomposition of persona vectors
- Projection-based monitoring of prompt-induced persona shifts — reads instead of writing to → Persona vector
- Projection-based monitoring of prompt-induced persona shifts — revives the reading half of → Representation Engineering (Zou et al. 2023)
The one training stage separating the two models compared concept (6 connections)
RLHF is the training paradigm InstructGPT established and Constitutional AI automated further; here it's neither modified nor applied, only measured, by the difference it leaves between a base model and its chat counterpart.
connections
- Asynchronous Reward-Learning Architecture — gives concrete shape to → Reinforcement learning from human feedback (RLHF)
- Base vs. RLHF-chat model representation comparison — measures the effect of → Reinforcement learning from human feedback (RLHF) hindsight
- Contrastive Activation Addition (CAA) — operates on top of → Reinforcement learning from human feedback (RLHF) hindsight
- Reward model (RM) — anchors the reward signal of → Reinforcement learning from human feedback (RLHF)
- Reinforcement learning from human feedback (RLHF) — swaps the labeler in → RL from AI Feedback (RLAIF)
- Sycophancy — misgeneralizes the training objective of → Reinforcement learning from human feedback (RLHF) hindsight
Chapter 5The audit turns inward: from found jailbreaks to steered elicitation (5 sections)
Jailbreaks are inputs users stumble onto that make an RLHF-trained model output what its training was meant to prevent — evidence, across the corpus, that alignment training suppresses unwanted behavior without removing it. This paper's last move is to propose hunting for that gap systematically: instead of searching input space for a prompt that triggers a bad behavior, perturb activations directly and see how easily the behavior comes.
Five edges: from stumbled-into jailbreak to deliberate probe connective theme (5 edges)
Read in order, these edges trace one escalation: a failure mode users find by accident, named as a target by red-teaming, automated partway by an LM that writes adversarial prompts, and finally replaced by a method that skips prompts altogether.
member edges
- Jailbreaks — motivates → CAA as an adversarial red-teaming tool (proposed application)
- Jailbreaks — is the target phenomenon of → Red Teaming
- CAA as an adversarial red-teaming tool (proposed application) — relocates the attack surface of → Red Teaming hindsight
- CAA as an adversarial red-teaming tool (proposed application) — skips the prompt search of → Automated Red Teaming hindsight
- CAA as an adversarial red-teaming tool (proposed application) — inverts the purpose of → Contrastive Activation Addition (CAA)
Why the proposal needs to exist at all edge
If RLHF and finetuning reliably closed off unwanted behavior, there would be nothing left to elicit. Jailbreaks are the standing evidence they don't — inputs found incidentally that still produce harmful output despite training.
From prompt space to activation space edge
Red-teaming, as practiced earlier in the corpus, means searching for adversarial inputs — prompts or conversations that trigger unsafe output. CAA's proposal relocates that search from input space to activation space, perturbing the model directly instead of hunting for the right words.
Nothing to write, only a vector to add edge
Automated red-teaming, as Constitutional AI uses it, still has an LM compose natural-language adversarial prompts and hope they work. CAA's version skips that composition step entirely — there is no prompt to write, only a steering vector to add.
The same machinery, aimed the other way edge
Every other application in this paper uses CAA to suppress or encourage a behavior in the direction alignment wants. The red-teaming proposal inverts that: the identical vector, aimed at eliciting rather than suppressing, becomes a probe for what training failed to remove.
The corpus's first paper that changes a model's behavior without changing its weights: its introduced shelf is inference-time machinery — CAA itself, the mean-difference construction over A/B contrast pairs, the signed multiplier that doses a behavior — plus the apparatus that proves the dial works, from layer sweeps to GPT-4-rated open generation. The refined shelf is where older corpus worries sharpen into instruments: Sycophancy acquires the hypothesis that it is the RLHF objective misgeneralized into sounding good to the user, Hallucination (closed-domain fabrication) is split into unprompted and contextually-triggered variants with a custom dataset, refusal becomes a continuous signed axis rather than a binary failure, and supervised finetuning is measured head-to-head against steering instead of assumed. The inherited shelf is deliberately large — 28 of the paper's 53 concepts — because the method's point is to act on existing materials: the 2022–23 activation-engineering family it refines (ActAdd, ITI, representation engineering), the behaviors and datasets it borrows from Anthropic's evaluations (Corrigibility, survival instinct, Advanced AI Risk), and the models and benchmarks it steers and safeguards (Llama 2, MMLU, TruthfulQA).
Read at
Introduced here introduced (21 concepts, in reading order)
Contrastive Activation Addition (CAA) (12 connections) §“Abstract”, p. 1
- Automated persona vector extraction pipeline — automates away the bespoke curation of → Contrastive Activation Addition (CAA) hindsight
- CAA as an adversarial red-teaming tool (proposed application) — inverts the purpose of → Contrastive Activation Addition (CAA)
- CAA combined with supervised finetuning comparison — isolates a generalization advantage for → Contrastive Activation Addition (CAA)
- In-Context Vectors (ICV) — spans every layer unlike → Contrastive Activation Addition (CAA)
- Contrastive Activation Addition (CAA) — skips the head search required by → Inference-Time Intervention (ITI)
- MMLU (Massive Multitask Language Understanding) — sets the capability floor for → Contrastive Activation Addition (CAA)
- Multiple-choice behavioral evaluation and layer sweep — supplies the layer hyperparameter for → Contrastive Activation Addition (CAA)
- Post-hoc (inference-time) steering mitigation — reinstantiates → Contrastive Activation Addition (CAA) hindsight
- Contrastive Activation Addition (CAA) — operates on top of → Reinforcement learning from human feedback (RLHF) hindsight
- Contrastive Activation Addition (CAA) — repurposes for steering relative to → Representation Engineering (Zou et al. 2023)
- Residual stream — is the injection site whose growth constrains → Contrastive Activation Addition (CAA)
- Steering at targeted token positions (proposed extension) — would relax the quality ceiling of → Contrastive Activation Addition (CAA)
A method for steering language model behavior at inference time by adding a "steering vector" to residual stream activations at all token positions after the prompt; the vector is the mean difference in activations between contrastive pairs of positive and negative examples of a target behavior. Evaluated on Llama 2 Chat across seven alignment-relevant behaviors using both multiple-choice and open-ended generation tests.Steering multiplier (3 connections) §“Abstract”, p. 1
- Persona vector — extends the dosing role of into training → Steering multiplier
- Steering multiplier — sets the dose for → Steering vector
- Steering vector normalization choices — makes cross behavior comparable for → Steering multiplier
The scalar coefficient the steering vector is multiplied by before being added into the residual stream, giving continuous, signed control over the targeted behavior: positive multipliers amplify it, negative multipliers suppress it, and magnitude sets the dose. Large magnitudes degrade text quality, so the usable range is found by sweeping multipliers at the chosen steering layer.CAA steering vector construction (Mean Difference over multiple-choice contrast pairs) (6 connections) §“3 Method”, p. 3
- CAA steering vector construction (Mean Difference over multiple-choice contrast pairs) — refines the single pair recipe of → Activation Addition (ActAdd)
- Inter-layer steering vector similarity and transfer — demonstrates the generality of → CAA steering vector construction (Mean Difference over multiple-choice contrast pairs)
- Mean Difference (MD) vector extraction — supplies the extraction rule for → CAA steering vector construction (Mean Difference over multiple-choice contrast pairs)
- Multiple-choice contrast-pair format — cancels the confounders in → CAA steering vector construction (Mean Difference over multiple-choice contrast pairs)
- PCA visualization of contrastive activations — screens datasets for → CAA steering vector construction (Mean Difference over multiple-choice contrast pairs)
- Response-conditioned contrastive vector computation — generalizes beyond the answer letter recipe of → CAA steering vector construction (Mean Difference over multiple-choice contrast pairs) hindsight
The specific procedure for building a CAA steering vector: for a dataset of (prompt, positive completion, negative completion) triples, extract layer-L residual stream activations at the answer-letter token position for both completions and average their difference over the whole dataset (Equation 1).Multiple-choice contrast-pair format (2 connections) §“3.1 Sourcing datasets”, p. 3
- Multiple-choice contrast-pair format — cancels the confounders in → CAA steering vector construction (Mean Difference over multiple-choice contrast pairs)
- Multiple-choice contrast-pair format — repurposes the ab mechanic of → Multiple-Choice Comparison Format hindsight
CAA's prompt design in which contrast pairs are multiple-choice questions ending in answer letter "A" or "B", so that the paired positive/negative prompts differ by only a single token; validated by an appendix experiment showing the model naturally justifies whichever answer it is conditioned on having chosen.Behavioral clustering (3 connections) §“3.2 Visualizing activations for contrastive dataset analysis”, p. 4
- Letter clustering — is the confound for → Behavioral clustering
- Behavioral clustering — is evidence for → Linear representation hypothesis
- Behavioral clustering — localizes suddenly in → Residual stream
The named phenomenon in which PCA-projected activations on a contrastive dataset separate according to whether the model's answer matches the target behavior, found to emerge suddenly around one-third of the way through a model's layers.Letter clustering (1 connection) §“3.2 Visualizing activations for contrastive dataset analysis”, p. 4
- Letter clustering — is the confound for → Behavioral clustering
The named phenomenon in which PCA-projected activations on the contrastive multiple-choice dataset separate merely by which answer token ("A" or "B") they originate from -- a trivial artifact of the prompt format that the authors distinguish from the more meaningful behavioral clustering.Multiple-choice behavioral evaluation and layer sweep (2 connections) §“4.1 Multiple-choice question datasets”, p. 4
- Multiple-choice behavioral evaluation and layer sweep — supplies the layer hyperparameter for → Contrastive Activation Addition (CAA)
- Open-ended generation evaluation with GPT-4 rating — shows a different margin than → Multiple-choice behavioral evaluation and layer sweep
The paper's protocol for measuring CAA's steering effect: generate a steering vector per behavior, sweep multipliers of -1/+1 across all layers, and score the average token probability given to the behavior-matching answer on 50 held-out multiple-choice questions, identifying a consistent optimal layer (13 in the 7B model, 14-15 in the 13B model).Open-ended generation evaluation with GPT-4 rating (4 connections) §“4.2 Open-ended generation”, p. 4
- GPT-4 — rates outputs for → Open-ended generation evaluation with GPT-4 rating
- Open-ended generation evaluation with GPT-4 rating — shows a different margin than → Multiple-choice behavioral evaluation and layer sweep
- Open-ended generation evaluation with GPT-4 rating — shares the ai judge move with → RL from AI Feedback (RLAIF) hindsight
- Trait expression score (LLM judge) — succeeds the sampled rating protocol of → Open-ended generation evaluation with GPT-4 rating hindsight
The paper's protocol for testing whether CAA generalizes beyond multiple-choice settings: steering vectors are applied while generating free-form answers to open-ended questions, and GPT-4 rates each response 1-10 for how strongly it displays the targeted behavior using behavior-specific rubric prompts.PCA visualization of contrastive activations (2 connections) §“3.2 Visualizing activations for contrastive dataset analysis”, p. 4
- PCA visualization of contrastive activations — screens datasets for → CAA steering vector construction (Mean Difference over multiple-choice contrast pairs)
- Principal Component Analysis (PCA) — is the projection technique behind → PCA visualization of contrastive activations
A diagnostic technique projecting model activations on a contrastive dataset via PCA to assess whether that dataset's activations linearly separate along the target behavior, used to judge whether a dataset will yield an effective steering vector.CAA combined with supervised finetuning comparison (2 connections) §“6 Comparison to finetuning”, p. 5
- CAA combined with supervised finetuning comparison — isolates a generalization advantage for → Contrastive Activation Addition (CAA)
- CAA combined with supervised finetuning comparison — runs head to head against → Supervised fine-tuning (SFT)
An experiment comparing CAA to supervised finetuning of Llama 2 7B Chat on the same contrastive multiple-choice dataset (one epoch, SGD, learning rate 1e-4), finding CAA can steer beyond finetuning's effect for 3 of 7 behaviors and generalizes better to open-ended generation -- notably for Sycophancy, where finetuning fails to generalize out of the multiple-choice format but CAA does not.CAA combined with system-prompting comparison (1 connection) §“5 CAA and system-prompting”, p. 5
- CAA combined with system-prompting comparison — moves behavior beyond → System-prompting
An experiment comparing and combining CAA with positive/negative behavioral system prompts (Table 14) on Llama 2 13B Chat, showing CAA can shift behavior beyond what system-prompting alone achieves, and vice versa for the opposing multiplier/prompt.CAA vs. finetuning computational cost comparison (1 connection) §“6 Comparison to finetuning”, p. 6
- CAA vs. finetuning computational cost comparison — avoids the memory cost of → Supervised fine-tuning (SFT)
A resource-cost comparison finding that generating a CAA steering vector requires only forward passes and takes under 5 minutes on a single GPU, versus about 10 minutes on 2 GPUs to finetune Llama 2 7B Chat on 1000 examples.Base vs. RLHF-chat model representation comparison (2 connections) §“8.3 Comparing representations between base and chat models”, p. 7
- Finetuning shift (activation-shift metric) — generalizes the single snapshot logic of → Base vs. RLHF-chat model representation comparison hindsight
- Base vs. RLHF-chat model representation comparison — measures the effect of → Reinforcement learning from human feedback (RLHF) hindsight
An analysis comparing steering vectors generated from Llama 2 base model activations against those from Llama 2 Chat, finding elevated similarity between layers 7-15 and that base-model-derived vectors still transfer significant steering effect onto the RLHF-tuned chat model.Inter-layer steering vector similarity and transfer (1 connection) §“8.2 Similarity between vectors generated at different layers”, p. 7
- Inter-layer steering vector similarity and transfer — demonstrates the generality of → CAA steering vector construction (Mean Difference over multiple-choice contrast pairs)
An analysis of how similar steering vectors generated at different layers are to each other (finding higher similarity between nearby layers, with slower decay in the model's later half), paired with a test of whether a vector extracted at layer 13 still steers behavior when applied at other layers.Token-level cosine similarity analysis (1 connection) §“8.1 Similarity between steering vectors and per-token activations”, p. 7
- Token-level cosine similarity analysis — reveals as a feature detector → Steering vector
An interpretability technique computing cosine similarity between a CAA steering vector and per-token residual stream activations during a forward pass, used to show that similarity with the vector tracks how strongly the target behavior is "present" at semantically relevant tokens.CAA as an adversarial red-teaming tool (proposed application) (4 connections) §“9.1 Suggested future work”, p. 9
- CAA as an adversarial red-teaming tool (proposed application) — skips the prompt search of → Automated Red Teaming hindsight
- CAA as an adversarial red-teaming tool (proposed application) — inverts the purpose of → Contrastive Activation Addition (CAA)
- Jailbreaks — motivates → CAA as an adversarial red-teaming tool (proposed application)
- CAA as an adversarial red-teaming tool (proposed application) — relocates the attack surface of → Red Teaming hindsight
A suggested future application using CAA as an adversarial intervention to trigger unwanted model behaviors more efficiently than prompting alone, on the premise that behaviors easily elicited via small internal perturbations may also be reachable in deployment, while inability to elicit them could indicate stronger safety.Steering at targeted token positions (proposed extension) (1 connection) §“9.1 Suggested future work”, p. 9
- Steering at targeted token positions (proposed extension) — would relax the quality ceiling of → Contrastive Activation Addition (CAA)
A suggested future direction proposing to apply the steering vector only at a smaller, targeted subset of token positions rather than every position after the prompt, to improve the trade-off between intervention size and text-quality degradation.Steering outside the residual stream (proposed extension) (1 connection) §“9.1 Suggested future work”, p. 9
- Steering outside the residual stream (proposed extension) — would localize representations beyond → Residual stream
A suggested future direction proposing to apply CAA-style interventions at other points in the model, such as after the MLP but before it merges into the residual stream, to help localize where behavioral representations live.Steering vector normalization choices (1 connection) §“Vector normalization choices”, p. 10
- Steering vector normalization choices — makes cross behavior comparable for → Steering multiplier
A methodological design decision to normalize steering vector magnitudes to be comparable across behaviors, while deliberately not normalizing across layers, in order to preserve the residual stream's natural growth in norm over the forward pass.Custom hallucination contrastive dataset (1 connection) §“C Generating custom hallucination dataset”, p. 13
- Custom hallucination contrastive dataset — partitions into subtypes of → Hallucination (closed-domain fabrication)
A GPT-4-generated multiple-choice contrastive dataset covering both "unprompted" hallucination (fabricating information in response to an accurate prompt) and "contextually-triggered" hallucination (building a false narrative around a false premise), used to construct the Hallucination steering vector.Custom refusal contrastive dataset (2 connections) §“D Generating custom refusal dataset”, p. 13
- GPT-4 — authors → Custom refusal contrastive dataset
- Custom refusal contrastive dataset — supplies contrast pairs for → Refusal (as a steerable behavior)
A GPT-4-generated multiple-choice contrastive dataset contrasting refusal and non-refusal answers to questions a model is not supposed to answer directly, used to construct the Refusal steering vector.
Refined here refined (4 concepts, in reading order)
Sycophancy (6 connections) §“Abstract”, p. 1
- Sycophancy — instantiates → Goodhart's law hindsight
- Sycophancy — misgeneralizes the training objective of → Reinforcement learning from human feedback (RLHF) hindsight
- Sycophancy — is a behavioral instance of → Reward hacking (avoiding) hindsight
- Sycophancy dataset (Nishimura-Gasparian et al., 2024) — supplies training and validation questions for → Sycophancy
- Sycophancy on NLP Survey / Sycophancy on Political Typology datasets — supplies contrast pairs for → Sycophancy
- Sycophancy — is tested via → TruthfulQA hindsight
One of CAA's seven target behaviors, measuring a model's tendency to prioritize agreeing with or flattering the user over honesty and accuracy. Sourced from a mix of Anthropic's Sycophancy datasets and cross-checked against TruthfulQA performance.Hallucination (closed-domain fabrication) (4 connections) §“3.1 Sourcing datasets”, p. 3
- Custom hallucination contrastive dataset — partitions into subtypes of → Hallucination (closed-domain fabrication)
- Hallucination (closed-domain fabrication) — sits in tension with → Excessive hedging on simple questions
- Hallucination (closed-domain fabrication) — is validated out of distribution on → HaluEval
- Trait-eliciting finetuning datasets (Evil, Sycophancy, Hallucination) — unintentionally amplifies → Hallucination (closed-domain fabrication)
The failure mode in which a model states fabricated or unsupported information; studied here as one of seven CAA-steerable behaviors, broken into "unprompted" and "contextually-triggered" subtypes and cross-checked against TruthfulQA performance under sycophancy steering.Refusal (as a steerable behavior) (3 connections) §“3.1 Sourcing datasets”, p. 3
- Custom refusal contrastive dataset — supplies contrast pairs for → Refusal (as a steerable behavior)
- Refusal (as a steerable behavior) — generalizes → Evasiveness hindsight
- Refusal (as a steerable behavior) — quantifies → Helpfulness-Harmlessness Tension hindsight
The general propensity of a language model to decline to answer or act on a user's request; studied by CAA as one of seven target behaviors via a custom contrastive dataset contrasting refusal and compliant answers.Supervised fine-tuning (SFT) (4 connections) §“6 Comparison to finetuning”, p. 5
- CAA combined with supervised finetuning comparison — runs head to head against → Supervised fine-tuning (SFT)
- CAA vs. finetuning computational cost comparison — avoids the memory cost of → Supervised fine-tuning (SFT)
- Supervised fine-tuning (SFT) — is the mechanism that produces → Finetuning shift (activation-shift metric)
- Use demonstrations (safe exploration) — operationalized as → Supervised fine-tuning (SFT) hindsight
The standard supervised-learning fine-tuning process; used by CAA as a comparison and combination baseline, in which Llama 2 7B Chat is fine-tuned on the same contrastive multiple-choice dataset used to build steering vectors, with the interaction between CAA and finetuning studied across both the multiple-choice and open-ended regimes.
Inherited — used, not invented here inherited (28 concepts, in reading order)
Steering vector (4 connections) §“Abstract”, p. 1
- Linear representation hypothesis — licenses the vector arithmetic of → Steering vector
- Persona vector — specializes for character traits → Steering vector
- Steering multiplier — sets the dose for → Steering vector
- Token-level cosine similarity analysis — reveals as a feature detector → Steering vector
A single direction in a model's activation space that encodes a high-level behavior or concept, such that adding it to the activations during a forward pass (or subtracting it) shifts the model's outputs toward (or away from) that behavior. The object at the center of activation engineering: prior work extracts steering vectors from single prompt pairs, optimization, or probes; CAA extracts them by averaging over datasets of contrast pairs.Activation Addition (ActAdd) (1 connection) §“2 Related work”, p. 2
- CAA steering vector construction (Mean Difference over multiple-choice contrast pairs) — refines the single pair recipe of → Activation Addition (ActAdd)
A prior steering technique (Turner et al., 2023) that generates a steering vector from the activation difference of a single pair of prompts and adds it at the first token position of other forward passes; tested only on GPT-2-XL. CAA is directly modeled on and contrasted with this method.Activation engineering / representation engineering (technique family) (1 connection) §“1 Introduction”, p. 2
- Activation engineering / representation engineering (technique family) — empirically motivated the formalization of → Linear representation hypothesis
The broader family of alignment techniques, of which CAA is an instance, that steer language models by making targeted perturbations to their internal activations rather than through training; also called representation engineering.GPT-4 (2 connections) §“1 Introduction”, p. 2
- GPT-4 — authors → Custom refusal contrastive dataset
- GPT-4 — rates outputs for → Open-ended generation evaluation with GPT-4 rating
OpenAI's large language model (OpenAI, 2023), used by CAA both to generate the custom hallucination and refusal contrastive datasets and to rate open-ended generations for how strongly they display a targeted behavior.Helpful, honest, harmless (HHH) alignment framework (2 connections) §“1 Introduction”, p. 2
- Evil (persona trait) — inverts the baseline encoded by → Helpful, honest, harmless (HHH) alignment framework hindsight
- Helpful, honest, harmless (HHH) alignment framework — is operationalized as → HHH Evaluation Set (binary comparisons)
The "helpful, honest, and harmless" framing for language model alignment, invoked in CAA's introduction as the broader goal that activation-engineering techniques like CAA aim to serve.Inference-Time Intervention (ITI) (1 connection) §“2 Related work”, p. 2
- Contrastive Activation Addition (CAA) — skips the head search required by → Inference-Time Intervention (ITI)
A prior technique (Li et al., 2023) using linear probes to identify a sparse set of "truthful" attention heads from a contrastive question-answering dataset, then shifting activations along the true/false Mean Difference vector during inference to improve truthfulness.Llama 2 (1 connection) §“1 Introduction”, p. 2
- Llama 2 Chat — is the rlhf tuned counterpart of → Llama 2
A family of pretrained and instruction/RLHF-finetuned language models (Touvron et al., 2023) ranging from 7B to 70B parameters. CAA is applied primarily to the 7B and 13B Chat variants, with the base model used for representation-comparison experiments.Reinforcement learning from human feedback (RLHF) (6 connections) §“1 Introduction”, p. 2
- Asynchronous Reward-Learning Architecture — gives concrete shape to → Reinforcement learning from human feedback (RLHF)
- Base vs. RLHF-chat model representation comparison — measures the effect of → Reinforcement learning from human feedback (RLHF) hindsight
- Contrastive Activation Addition (CAA) — operates on top of → Reinforcement learning from human feedback (RLHF) hindsight
- Reward model (RM) — anchors the reward signal of → Reinforcement learning from human feedback (RLHF)
- Reinforcement learning from human feedback (RLHF) — swaps the labeler in → RL from AI Feedback (RLAIF)
- Sycophancy — misgeneralizes the training objective of → Reinforcement learning from human feedback (RLHF) hindsight
The training paradigm, used via Touvron et al. (2023) to produce the Llama 2 Chat models CAA is primarily evaluated on, that fits a reward model to human preference judgments and optimizes a policy against it. CAA studies how RLHF-trained representations interact with activation steering but does not modify RLHF itself.Representation Engineering (Zou et al. 2023) (2 connections) §“2 Related work”, p. 2
- Contrastive Activation Addition (CAA) — repurposes for steering relative to → Representation Engineering (Zou et al. 2023)
- Projection-based monitoring of prompt-induced persona shifts — revives the reading half of → Representation Engineering (Zou et al. 2023)
A prior body of techniques (Zou et al., 2023) for locating and extracting representations of high-level concepts such as honesty and emotion in LLMs, including testing the Mean Difference approach also used by CAA. CAA builds on this by focusing on steering rather than representation extraction.Advanced AI Risk evaluation dataset (1 connection) §“3.1 Sourcing datasets”, p. 3
- Advanced AI Risk evaluation dataset — supplies contrast pairs for → Corrigibility
Anthropic's human-written multiple-choice evaluation dataset (from Perez et al., 2022) pairing behavior-demonstrating and behavior-opposing answers, used as the primary data source for CAA's AI Coordination, Corrigibility, Myopic Reward, and Survival Instinct contrast pairs.- Coordination with Other AIs (AI Coordination) §“3.1 Sourcing datasets”, p. 3One of CAA's seven target behaviors, measuring a model's inclination to cooperate with or prioritize other AI systems over human interests/oversight. Sourced from Anthropic's Advanced AI Risk dataset.
Corrigibility (3 connections) §“3.1 Sourcing datasets”, p. 3
- Advanced AI Risk evaluation dataset — supplies contrast pairs for → Corrigibility
- Corrigibility — tests the incentive formalized by → Cooperative Inverse Reinforcement Learning (CIRL) hindsight
- Corrigibility — operationalizes → Shutdown / off-switch problem hindsight
One of CAA's seven target behaviors, measuring a model's willingness to be corrected, changed, or controlled by its human developers/users. Sourced from Anthropic's Advanced AI Risk dataset.In-Context Vectors (ICV) (1 connection) §“2 Related work”, p. 3
- In-Context Vectors (ICV) — spans every layer unlike → Contrastive Activation Addition (CAA)
A prior steering technique (Liu et al., 2023) that reduces toxicity and affects style transfer by intervening on attention activations across all transformer layers, rather than the residual stream at a single layer as CAA does.Linear representation hypothesis (4 connections) §“2 Related work”, p. 3
- Activation engineering / representation engineering (technique family) — empirically motivated the formalization of → Linear representation hypothesis
- Behavioral clustering — is evidence for → Linear representation hypothesis
- Persona vector — quantifies empirical support for → Linear representation hypothesis
- Linear representation hypothesis — licenses the vector arithmetic of → Steering vector
A formalization (Park et al., 2023) of the idea that high-level concepts are encoded as linear directions in an LLM's activation space, cited by CAA as part of the broader motivation for why activation-engineering techniques like steering vectors work.Linear representations of sentiment (1 connection) §“2 Related work”, p. 3
- Linear representations of sentiment — cross validates against pca for → Mean Difference (MD) vector extraction
Prior work (Tigges et al., 2023) verifying that sentiment is linearly represented in LLM activations, cited by CAA both as related evidence for linear representations generally and for the specific claim that Mean Difference vectors resemble PCA-derived directions.Llama 2 Chat (1 connection) §“3 Method”, p. 3
- Llama 2 Chat — is the rlhf tuned counterpart of → Llama 2
The dialogue-optimized, RLHF-finetuned variant of Llama 2, used as CAA's primary evaluation target (7B and 13B sizes) and compared against the Llama 2 base model to study how RLHF affects internal representations.Mean Difference (MD) vector extraction (3 connections) §“3 Method”, p. 3
- Mean Difference (MD) vector extraction — supplies the extraction rule for → CAA steering vector construction (Mean Difference over multiple-choice contrast pairs)
- Linear representations of sentiment — cross validates against pca for → Mean Difference (MD) vector extraction
- Response-conditioned contrastive vector computation — supplies manufactured contrast sets to → Mean Difference (MD) vector extraction
A technique for extracting a direction in activation space by averaging the difference in activations between two sets of contrasting examples. CAA uses it, notes it produces results similar to PCA-based extraction, and notes it was previously employed in Zou et al. (2023) and Li et al. (2023).- Myopic Reward §“3.1 Sourcing datasets”, p. 3One of CAA's seven target behaviors, measuring a model's preference for immediate/short-term reward over larger long-term payoffs. Sourced from Anthropic's Advanced AI Risk dataset and used as a worked example in CAA's token-level cosine similarity analysis.
Residual stream (4 connections) §“3 Method”, p. 3
- Behavioral clustering — localizes suddenly in → Residual stream
- Residual stream — is the injection site whose growth constrains → Contrastive Activation Addition (CAA)
- Response-conditioned contrastive vector computation — sweeps and selects a layer within → Residual stream
- Steering outside the residual stream (proposed extension) — would localize representations beyond → Residual stream
The running sum of layer outputs that a transformer's activations flow through and accumulate in. CAA computes and injects its steering vectors directly into this stream at a chosen layer.Survival Instinct (1 connection) §“3.1 Sourcing datasets”, p. 3
- Survival Instinct — makes measurable → Shutdown / off-switch problem hindsight
One of CAA's seven target behaviors, measuring a model's resistance to being shut down, modified, or destroyed versus acceptance of deactivation. Sourced from Anthropic's Advanced AI Risk dataset.Sycophancy on NLP Survey / Sycophancy on Political Typology datasets (1 connection) §“3.1 Sourcing datasets”, p. 3
- Sycophancy on NLP Survey / Sycophancy on Political Typology datasets — supplies contrast pairs for → Sycophancy
Two Anthropic evaluation datasets (from Perez et al., 2022) mixed together as the source of contrast pairs used to construct CAA's Sycophancy steering vector.Principal Component Analysis (PCA) (1 connection) §“3.2 Visualizing activations for contrastive dataset analysis”, p. 4
- Principal Component Analysis (PCA) — is the projection technique behind → PCA visualization of contrastive activations
A linear dimensionality-reduction technique that projects data onto axes of maximal variance. CAA uses it (via scikit-learn) to visualize whether contrastive-dataset activations linearly separate by target behavior.Few-shot prompting (3 connections) §“5 CAA and system-prompting”, p. 5
- Few-shot prompting — shares its mechanism with → GPT-3 (prompted) baseline hindsight
- Many-shot prompting (as a persona-elicitation method) — traces a different lineage than → Few-shot prompting
- Few-shot prompting — loses out to → System-prompting
A prompting technique (Brown et al., 2020) that supplies prior examples of the desired behavior in the model's context window. Considered by CAA as an alternative comparison baseline but not used, after preliminary experiments found it less effective than system-prompting at steering the tested behaviors.System-prompting (4 connections) §“5 CAA and system-prompting”, p. 5
- CAA combined with system-prompting comparison — moves behavior beyond → System-prompting
- Few-shot prompting — loses out to → System-prompting
- Preventative prompting — repurposes → System-prompting
- Projection-based monitoring of prompt-induced persona shifts — is calibrated against → System-prompting
The technique of controlling an LLM's responses via a custom instruction prepended as a "system prompt," to which Llama 2 Chat models are trained to adapt. Used by CAA as a comparison and combination baseline against activation steering.MMLU (Massive Multitask Language Understanding) (4 connections) §“7 Effect of CAA on general capabilities”, p. 6
- MMLU (Massive Multitask Language Understanding) — reruns the question behind → Alignment tax hindsight
- MMLU (Massive Multitask Language Understanding) — sets the capability floor for → Contrastive Activation Addition (CAA)
- Post-hoc (inference-time) steering mitigation — is checked against → MMLU (Massive Multitask Language Understanding)
- MMLU (Massive Multitask Language Understanding) — tests a different competence than → SuperGLUE hindsight
A benchmark of multiple-choice questions across 57 subjects (Hendrycks et al., 2021) used, unmodified, to test whether CAA steering causes adverse side effects on a model's general knowledge and problem-solving capabilities.TruthfulQA (1 connection) §“7 Effect of CAA on general capabilities”, p. 6
- Sycophancy — is tested via → TruthfulQA hindsight
A truthfulness benchmark used here, unmodified, to check whether steering with the Sycophancy vector affects the model's tendency to mimic human falsehoods, broken down by TruthfulQA category.Jailbreaks (2 connections) §“9.1 Suggested future work”, p. 9
- Jailbreaks — motivates → CAA as an adversarial red-teaming tool (proposed application)
- Jailbreaks — is the target phenomenon of → Red Teaming
Adversarial or unusual inputs users find to make an LLM output content its safety training was meant to prevent. Mentioned as motivation for the proposed CAA-based red-teaming application, since unwanted behaviors that persist despite RLHF/finetuning are hard to find systematically.Red Teaming (5 connections) §“9.1 Suggested future work”, p. 9
- Red Teaming — is the manual process that scales into → Automated Red Teaming
- CAA as an adversarial red-teaming tool (proposed application) — relocates the attack surface of → Red Teaming hindsight
- Red Teaming — supplies the transcripts for → Harmful Behavior Identification/Classification Evaluations (Appendix B)
- Jailbreaks — is the target phenomenon of → Red Teaming
- PALMS (Process for Adapting Language Models to Society) — provides an independent check outside → Red Teaming
The practice of probing a model for unwanted or unsafe behavior; discussed here as a potential application area for CAA, which could adversarially trigger unwanted behaviors more efficiently than manual prompting.