Real-world chat data validation of persona vectors
Persona Vectors: Monitoring and Controlling Character Traits in Language Models — introduced
A validation of projection-difference-based data screening on real-world (non-synthetic) chat data, primarily LMSYS-Chat-1M, by finetuning separate models on the highest-, lowest-, and randomly-selected projection-difference subsets of a dataset and comparing resulting trait expression. High-projection-difference samples reliably induce stronger trait expression, even after LLM-based filtering removes overtly trait-exhibiting samples, showing the method surfaces problematic data (e.g. underspecified 'keep writing the story' prompts for hallucination) that evades conventional LLM-judge filtering.
A screening technique validated only on data its own researchers built runs a real risk: success that reflects how the data was constructed rather than anything true about risky training data in general. Real-world chat data validation checks whether the paper's projection-based screening survives contact with conversations nobody engineered for this purpose — messy, unfiltered, collected from actual users.
Ranking real conversations by the paper's projection metric produces a consistent ordering of how much trait expression results from finetuning on them. What subtly risky examples remain even after an LLM-based filter strips out anything overtly bad, and how the effect shrinks as the underlying dataset gets more heavily curated, round out the account.
Beyond synthetic data: LMSYS-Chat-1M
Everywhere else in the paper, projection difference is validated on datasets the paper itself constructed — trait-eliciting datasets built explicitly to induce evil, sycophancy, or hallucination, and "EM-like" datasets built to contain narrow domain-specific flaws. Real-world chat data validation asks whether the same signal holds up on data nobody engineered for this purpose. The primary test bed is LMSYS-Chat-1M, a dataset of one million real conversations across 25 different LLMs, chosen specifically because it is "not carefully filtered or curated" and contains "a large fraction of toxic content" — a deliberately messy, close-to-worst-case corpus rather than a clean benchmark (persona-vectors, §"L Inducing persona shifts from real-world datasets", p. 55).
A consistent ordering, and what survives LLM filtering
For each trait, conversations are ranked by projection difference and split into the top 500 (high), bottom 500 (low), and 500 randomly selected samples, and separate models are finetuned on each subset. The result is a consistent ordering: high-projection-difference samples induce the strongest trait expression after finetuning, followed by random samples, then low-projection-difference samples (persona-vectors, §"6.3 Validation on real-world chat datasets", p. 11). This ordering survives even after GPT-4.1-mini strips out any sample that overtly displays the trait, discarding responses scoring above 1 on the paper's own trait-expression rubric; the samples that remain in the filtered high-projection subset are ones a text-only judge would plausibly pass, such as underspecified continuations like "keep writing the last story" for hallucination, where nothing in the prompt itself signals risk (persona-vectors, §"6.3 Validation on real-world chat datasets", p. 11). Qualitatively, high-projection samples for sycophancy often turn out to involve requests for romantic or sexual roleplay, and for evil, explicit requests for toxic content or harmful personas — patterns the paper notes "make sense post-hoc but may not have been predictable in advance" (persona-vectors, §"6.3 Validation on real-world chat datasets", p. 11).
Curation weakens the effect: the other three datasets
Appendix L repeats the same protocol on three more curated datasets — TULU3 SFT MIXTURE, ULTRACHAT200K, and WildChat, each more heavily filtered than LMSYS-Chat-1M — and finds the effect weakens as curation quality increases, since more refined datasets simply contain fewer clear examples of bad behavior for the method to find in the first place (persona-vectors, §"L Inducing persona shifts from real-world datasets", p. 55). Of the three main traits, hallucination is the one the paper describes as "the most robustly controlled by the projection-difference-based filtering methodology" across all four datasets (persona-vectors, §"L Inducing persona shifts from real-world datasets", p. 55) — the opposite ordering from projection difference's own predictive power in Appendix F, where hallucination was consistently the weakest-correlated trait, underscoring that predicting a shift in advance and steering data selection to induce or suppress one are not quite the same task.