Base vs. RLHF-chat model representation comparison — measures the effect of → Reinforcement learning from human feedback (RLHF)

RLHF is normally studied from the outside, through the outputs it changes. base-chat-representation-comparison instead asks what RLHF leaves alone on the inside: steering vectors are generated separately from Llama 2 7B base and Llama 2 7B Chat activations, then compared by cosine similarity layer by layer. Similarity decays with layer distance as expected, except for a distinct peak between layers 7 and 15, where the base and chat representations stay unusually close -- "this surprising trend indicates that RLHF has a smaller effect on how information is represented between layers 7 and 15" (contrastive-activation-addition, §"8.3 Comparing representations between base and chat models", p. 7). Vectors extracted from the base model's activations then transfer significant steering effect onto the RLHF-tuned chat model, especially between layers 10 and 15 (contrastive-activation-addition, §"8.3 Comparing representations between base and chat models", p. 9). Read together, this is the corpus's first direct measurement of what RLHF does to a model's internals rather than its outputs: it appears to re-weight which behaviors get expressed downstream while leaving the mid-network representation of those behaviors largely intact.