Base versus chat: what RLHF changes inside
Reinforcement learning from human feedback changes a model's behavior dramatically; an RLHF-tuned chat model answers very differently than the same weights before that training. But it is a separate question whether it changes how the model represents things internally, or just which of its existing representations get expressed. This theme puts that question to a direct test, using the base and chat versions of the same model as a natural before-and-after.
The page covers the comparison at its center: steering vectors extracted from the base model and from the chat model turn out to be unusually similar in the network's middle layers, and a vector built on the base model still works when applied to the chat one. Since RLHF is the only thing separating the two, the result reads as a finding about RLHF itself, that it reweights which behaviors surface more than it rewrites how they are encoded.
Llama 2 is Touvron et al.'s family of pretrained and RLHF-finetuned models, ranging from 7 to 70 billion parameters, and the paper works with two of its variants specifically to isolate what reinforcement learning from human feedback changes inside the network. RLHF is the training paradigm in question: fitting a reward model to human preference judgments and optimizing a policy against it, used by Touvron et al. to produce Llama 2 Chat, the dialogue-optimized variant CAA steers throughout the rest of the paper. Here, though, Llama 2 Chat is compared against its own pretrained base rather than steered directly. The base-versus-chat representation comparison generates steering vectors from base-model activations and from chat-model activations and finds their cosine similarity is elevated specifically between layers 7 and 15, and, further, that vectors extracted from the base model still transfer significant steering effect when applied to the chat model. Since RLHF is the only thing separating base from chat, the finding reads as a measurement of RLHF itself: it reweights which behaviors a model expresses without substantially rewriting how those behaviors are represented in the mid-network layers, at least as far as steering vectors can see.