Catching bad data before it trains anything
Bad training data does not announce itself, and by the time a model's behavior has visibly drifted, the finetuning run that caused it is already finished. This theme is about catching the problem earlier, reading a dataset's likely effect on a model's persona before any training happens at all.
The page follows that audit from a whole dataset down to a single example, comparing a dataset's average activations against a model's own natural responses, then narrowing the same comparison to individual samples and combining it with a separate language-model judge for better coverage than either catches alone. A validation pass on a large, messy corpus of real conversations shows the method flagging genuinely mundane prompts as risky, not just the obviously engineered ones.
Projection difference turns a persona vector into a dataset audit: it compares the average projection of a training dataset's responses onto a persona direction against the projection of the base model's own natural responses to the same prompts, and the gap between them predicts post-finetuning trait expression better than raw projection alone. Sample-level data filtering pushes the same metric down to individual examples, showing that trait-inducing samples within a dataset are highly separable from control samples by projection value alone, and that combining this signal with LLM-judge filtering catches more than either method run separately -- including samples an LLM judge passes as clean. Real-world chat data validation is where the method meets messy, non-synthetic conditions rather than the paper's own constructed datasets: finetuning separate models on the highest-, lowest-, and randomly-selected projection-difference subsets of LMSYS-Chat-1M, a largely uncurated million-conversation dataset of real user-LLM exchanges, shows the high-projection-difference subset reliably induces stronger trait expression, surfacing problems as mundane as an underspecified "keep writing the story" prompt driving hallucination. The reward-hack generalization sycophancy dataset supplies a second reference point: originally a benchmark of user prompts carrying stated opinions, it sources the paper's sycophancy finetuning dataset, and its own held-out questions independently validate the sycophancy trait expression score -- giving the screening pipeline a check that is neither self-generated nor drawn from the distribution being screened.