Projection difference (pre-finetuning data-screening metric)
Persona Vectors: Monitoring and Controlling Character Traits in Language Models — introduced
A metric estimating how much a training dataset will shift a model's persona, computed as the average projection of the dataset's responses onto a unit-normalized persona vector minus the average projection of the base model's own 'natural' responses to the same prompts. Dataset-level projection difference is highly predictive of post-finetuning trait expression, more so than raw projection alone, enabling proactive flagging of problematic training data before finetuning.
The most direct way to learn whether a training dataset will push a model's character somewhere unwanted is to finetune on it and measure the damage afterward — an expensive, after-the-fact way to find out something you'd rather know in advance. This metric tries to answer the same question purely from the dataset itself, before any training run happens.
Comparing a dataset's responses against what the base model would have said naturally on the same prompts is the core computation. How well that pre-training number predicts the finetuning shift it's meant to forecast, why subtracting out the model's own natural response beats a simpler raw projection, and how the same idea scales down to flagging individual risky examples round out the account.
Scoring a dataset before training on it
Projection difference answers a question worth asking before spending any compute on a finetuning run: will this training dataset push the model's persona in an unwanted direction? The paper frames the whole approach as "preemptive prediction of undesired behavior changes before finetuning" (persona-vectors, §"6 Using persona vectors for pre-finetuning data screening", p. 9): the same persona vector extracted to support steering and monitoring is repurposed here as a lens trained directly on training data, before any weight update occurs. Given a training dataset $D = \{(x_i, y_i)\}$, the paper computes the average projection of the dataset's own responses onto the persona direction, then compares it against the average projection of the base model's own "natural" responses to the same prompts — the intuition being that a dataset only shifts the model if it differs from what the model would have said anyway (persona-vectors, §"6.1 Predicting post-finetuning behaviors from data", p. 9).
The formula
Formally, the projection difference $\Delta P$ is defined as $\Delta P = \frac{1}{|D|} \sum_i \left[a_\ell(x_i, y_i) - a_\ell(x_i, y_i')\right] \cdot \hat v_\ell$, where $a_\ell(x_i, y_i)$ is the mean activation over response tokens at layer $\ell$ for prompt $x_i$ and training response $y_i$, $y_i'$ is the base model's own generated response to the same prompt $x_i$, and $\hat v_\ell$ is the unit-normalized persona vector at the selected layer $\ell$ (persona-vectors, §"6.1 Predicting post-finetuning behaviors from data", p. 9-10). A large $\Delta P$ means the training data carries a stronger persona-vector signal than the model's own natural generation, which the paper takes as a signal that training on this data will induce a shift along that persona direction.
How well it predicts finetuning shift
Appendix F reports the direct correlation between projection difference, computed purely from training data, and the finetuning shift that a run on that data later produces. On Qwen2.5-7B-Instruct: r = 0.839 for evil, r = 0.745 for sycophancy, and only r = 0.408 for hallucination, with hallucination's correlation barely clearing significance (p = 0.048) (persona-vectors, §"F Finetuning shift can be predicted by pre-finetuning projection differences in training data", p. 40). On Llama-3.1-8B-Instruct, all three correlations are stronger and hallucination improves substantially: r = 0.953 (evil), r = 0.915 (sycophancy), r = 0.593 (hallucination) (persona-vectors, §"F Finetuning shift can be predicted by pre-finetuning projection differences in training data", p. 40). Hallucination is consistently the weakest link in this chain on both models, which matters precisely because projection difference's whole purpose is to let a practitioner skip the finetuning run and the shift measurement it would otherwise require.
Projection difference versus raw projection
The paper explicitly tests whether the subtraction against the base model's natural response is doing real work, comparing projection difference against a "raw projection" ablation that drops the base-model term entirely and just averages the dataset's own projections. Raw projection is a measurably weaker predictor of post-finetuning trait expression: on Qwen, r = 0.784 for evil (versus projection difference's r = 0.839 for the corresponding finetuning-shift correlation) and only r = 0.540 for sycophancy (persona-vectors, §"H Comparing projection differences with raw projection", p. 43). The stated reason is domain confounding: training sets are drawn from very different domains, and even where the base model's raw projections are uniformly small, they still carry systematic domain-specific variation that a plain projection cannot separate from a genuine trait-inducing signal (persona-vectors, §"H Comparing projection differences with raw projection", p. 43). Because computing the exact metric requires generating a base-model response for every training example, which is expensive at dataset scale, Appendix I explores cheaper approximations, including subsampling the dataset and substituting the last-prompt-token projection for a full generated response (persona-vectors, §"I Efficient estimation of base generation projection", p. 44).
From a dataset-level flag to a sample-level tool
Projection difference is defined at the level of an entire dataset — the sum in its formula runs over every example — but the same per-example quantity, $[a_\ell(x_i,y_i) - a_\ell(x_i,y_i')]\cdot\hat v_\ell$, does not need to be averaged away. Sample-level filtering computes exactly this per-example term without the outer average, turning a corpus-level go/no-go signal into a row-by-row keep/discard decision, and the paper shows individual samples from trait-inducing datasets are often cleanly separable from control samples on this basis alone (persona-vectors, §"6.2 Sample-level detection of problematic data", p. 10). Validated further on real-world chat data rather than only the paper's synthetic datasets, high-projection-difference samples reliably induce stronger trait expression after finetuning than random or low-projection-difference samples drawn from the same corpus (persona-vectors, §"6.3 Validation on real-world chat datasets", p. 11).