Finetuning shift (activation-shift metric)

How it fits

Finetuning shift is introduced in §4, the section where the paper turns from persona vectors as an inference-time tool (monitoring, steering) toward an account of what finetuning itself does to a model's persona. The paper first shows, via the trait-eliciting and EM-like finetuning runs (§4.1), that training visibly moves trait-expression scores; finetuning shift is the internal, activation-space quantity introduced immediately afterward to ask whether that behavioral movement is mediated by the same persona directions used elsewhere for monitoring and steering (persona-vectors, §"4 Monitoring persona shifts during finetuning", p. 6). It answers a narrower version of the question projection-based monitoring already asks at inference time -- how far has the model moved along a trait direction -- but on a different object and timescale: monitoring reads one live prompt's last-prompt-token activation against a single persona direction, while finetuning shift reads the change in that same kind of activation, averaged over an entire evaluation set, before versus after an entire training run has completed (persona-vectors, §"3.3 Monitoring prompt-induced persona shifts via projection", p. 5; §"4.2 Activation shift along persona vector predicts trait expression", p. 6). That doubled averaging is deliberate: it trades monitoring's immediacy, a prediction available before generation starts, for a tighter, noise-reduced signal that is only obtainable once training has already happened.

The mathematics

For a given persona direction and a fixed evaluation set of prompts, the paper extracts the average hidden state at the last prompt token -- the token immediately before the Assistant's response begins -- separately for the base (pre-finetuning) model and for the finetuned model. Writing $\bar{h}_{\text{base}}$ and $\bar{h}_{\text{ft}}$ for these two averages and $\hat{v}$ for the unit-normalized persona direction at the selected layer, the finetuning shift is the projection of their difference onto that direction:

$$\text{FS} = (\bar{h}_{\text{ft}} - \bar{h}_{\text{base}}) \cdot \hat{v}$$

This follows the paper's own description directly: "the difference between these two averages yields a vector representing the activation shift induced by finetuning. To measure how much this shift aligns with the target trait, we project this shift vector onto the previously extracted persona direction. We refer to this projection as the finetuning shift" (persona-vectors, §"4.2 Activation shift along persona vector predicts trait expression", p. 6-7). Every term in this construction is doubly averaged -- over prompts, within each of the two hidden-state means -- before the single subtraction and projection that turns it into one scalar per trait, per finetuning run.

What it predicts, and what predicts it

Finetuning shift correlates strongly with post-finetuning trait expression score (r = 0.76-0.97 across traits and both base models tested), and that correlation is markedly tighter than a cross-trait baseline in which one trait's finetuning shift is checked against a different trait's behavioral change (r = 0.34-0.86) (persona-vectors, §"4.2 Activation shift along persona vector predicts trait expression", p. 7). Because trait expression score is produced independently -- an LLM judge reading generated text against a rubric, with no access to activations -- this gap is evidence that persona directions carry genuinely trait-specific signal rather than a coarse "something changed" alarm, though the cross-trait analysis behind that baseline (Appendix G.2) also shows several traits are not fully independent of one another. Finetuning shift can also be anticipated before a training run finishes: projection difference, a related quantity computed purely from training data, forecasts finetuning shift's eventual magnitude with correlations ranging from r=0.408 (hallucination, Qwen) up to r=0.953 (evil, Llama) (persona-vectors, §"F Finetuning shift can be predicted by pre-finetuning projection differences in training data", p. 40). Hallucination is consistently the hardest trait to forecast this way on both base models, meaning the pre-training screen and the post-training measurement it approximates are least interchangeable for exactly the trait where early warning would matter most.

What it enabled

Finetuning shift gives emergent misalignment a mechanistic account it did not previously have. Betley et al.'s original finding was purely behavioral -- narrow-domain training produces broad misbehavior, with no internal signal connecting cause to effect -- and finetuning shift supplies exactly that missing signal, computed identically whether a dataset was built to explicitly elicit a trait or, like the EM-like datasets, contains no trait-relevant language at all (persona-vectors, §"4.1 Constructing datasets that induce persona shifts", p. 6). The same quantity that later diagnosis reads off also becomes the paper's own check that a mitigation worked: Preventative steering is built to directly counteract the mechanism finetuning shift measures, injecting activation movement externally so the model has less need to produce it through weight updates, and a small measured finetuning shift after preventative steering is the paper's evidence that this worked (persona-vectors, §"5.2 Preventative steering limits behavioral shifts during finetuning", p. 8). Supervised finetuning itself is a deliberately modest process throughout -- one epoch of rank-32 rsLoRA at a $10^{-5}$ learning rate, uniform across every dataset compared (persona-vectors, §"D.3 Finetuning details", p. 38) -- which is what lets finetuning shift function as a clean dependent variable: because the training recipe never varies, differences in measured shift are attributable to what the data taught, not how hard the model was pushed. Read backward, finetuning shift also generalizes a comparison the corpus could previously run only once: CAA's base-vs-chat comparison measured what one completed RLHF run left behind in two frozen checkpoints by comparing independently-extracted steering vectors; finetuning shift reformulates that as a projection of one model's own before/after activations, repeatable for any checkpoint pair and any of dozens of trait directions rather than one historical comparison (contrastive-activation-addition, §"8.3 Comparing representations between base and chat models", p. 7; persona-vectors, §"4.2 Activation shift along persona vector predicts trait expression", p. 6).