Base versus chat: what RLHF changes inside — supplies a mechanism for → Where RLHF alignment still falls short

Where RLHF alignment still falls short catalogs failures, hallucination, hedging, reward over-optimization, without saying why training leaves them reachable. The 2023 paper's base-versus-chat comparison supplies a candidate mechanism. Steering vectors extracted from Llama 2's pretrained base model and from its RLHF-trained chat counterpart show cosine similarity elevated specifically between layers 7 and 15, and vectors built purely from base-model activations still transfer significant steering effect onto the chat model in that same range (contrastive-activation-addition, §"8.3 Comparing representations between base and chat models", p. 8). If RLHF reweighted behavior expression without rewriting mid-network representations, a failure mode like sycophancy, present in an entirely different RLHF-trained model than InstructGPT, would survive training not as a bug the pipeline missed but because the representation it rides on was never substantially altered, only suppressed, and so remains reachable by a small post-hoc intervention rather than requiring retraining to fix.