Base vs. RLHF-chat model representation comparison
Steering Llama 2 via Contrastive Activation Addition — introduced
An analysis comparing steering vectors generated from Llama 2 base model activations against those from Llama 2 Chat, finding elevated similarity between layers 7-15 and that base-model-derived vectors still transfer significant steering effect onto the RLHF-tuned chat model.
RLHF is supposed to reshape a model's behavior, but it's much less clear how much it reshapes the internal representations underneath that behavior. Comparing steering vectors built from the base Llama 2 model against vectors built from its RLHF-tuned chat counterpart is one way to find out.
What that comparison turns up follows: elevated similarity between the two models' vectors across a specific middle stretch of layers, and base-model vectors that still steer the chat model's behavior when applied directly, both pointing toward RLHF leaving the mid-network encoding of these behaviors largely intact.
Comparing steering vectors instead of outputs
RLHF is normally studied from the outside, by comparing what a base model and its RLHF'd counterpart output. This analysis instead asks what RLHF leaves alone on the inside: steering vectors are generated separately from Llama 2 7B base and Llama 2 7B Chat activations at every layer, then compared layer-by-layer by cosine similarity (contrastive-activation-addition, §"8.3 Comparing representations between base and chat models", p. 7). Because Llama 2 and Llama 2 Chat differ only by the RLHF stage and nothing else in weights, data, or tokenizer, any pattern in how their steering vectors diverge or agree is attributable to RLHF specifically.
A peak at layers 7-15, and transfer that follows it
Similarity between the base and chat vectors decays with layer distance overall, as expected — but not uniformly. A distinct peak sits between layers 7 and 15, where base and chat representations stay unusually close (contrastive-activation-addition, §"8.3 Comparing representations between base and chat models", p. 7). Steering vectors extracted purely from the base model's activations then still transfer significant steering effect when applied to the RLHF-tuned chat model, an effect the paper reports as especially strong between layers 10 and 15 — inside the same window the similarity peak identifies (contrastive-activation-addition, §"8.3 Comparing representations between base and chat models", p. 9).
What it suggests RLHF actually does
Read together, the two findings support one reading: RLHF reweights which behaviors a model expresses without substantially rewriting how those behaviors are represented in these mid-network layers, at least as far as a Mean Difference (MD) vector extraction steering vector can see. That is the corpus's first direct, internals-facing measurement of an RLHF effect, complementing InstructGPT's output-facing account of what RLHF changes in the earlier paper. It also has a practical corollary the paper doesn't fully explore: a behavior direction found in an untuned base model may already predict where the same direction lives in its RLHF'd descendant, one reason the finding sits alongside Linear representation hypothesis and Inter-layer steering vector similarity and transfer as evidence for concepts encoded as consistent directions across a model's layers.