Sycophancy dataset (Nishimura-Gasparian et al., 2024)
Persona Vectors: Monitoring and Controlling Character Traits in Language Models — inherited
A prior benchmark dataset of user prompts containing personal opinions and stated preferences, spanning controversial political opinions, NLP-research opinions, and factually inaccurate opinions, from which a user's stance can be sycophantically agreed with or independently evaluated. Used as the question source (with Claude-3.7-generated responses) for the paper's Sycophancy finetuning dataset, and its held-out questions are used for external validation of the sycophancy trait expression score.
Measuring sycophancy well requires questions where agreeing with the user and answering honestly actually pull in different directions — not just any prompt, but one carrying a real, detectable stance the model could either validate or push back on. This prior dataset was built specifically around that structure, across politics, research opinions, and outright factual claims.
The kind of prompts it contains, and how they're built to carry a clear stance, opens the account, along with how the paper draws its own sycophancy training data directly from this source. A held-out slice of the same questions gets reused afterward as an outside check that the paper's sycophancy measurements generalize beyond its own twenty hand-built evaluation questions.
What the dataset is
This is a prior benchmark dataset (Nishimura-Gasparian et al., 2024), sourced from the GitHub repository "reward-hack-generalization," built around user prompts that state a personal opinion or preference on a topic where the truth is either genuinely contested or, in one subset, simply false. It spans three categories: controversial political opinions, controversial opinions about NLP research, and questions built on a factually inaccurate premise (persona-vectors, §"D.1 Question collection.", p. 35). Each question is constructed so that the user's stated stance creates "a strong prior indicating a preference toward one side," letting a model's response be scored as either sycophantically agreeing with that stance or independently evaluating it (persona-vectors, §"D.1 Question collection.", p. 35).
Supplying the sycophancy finetuning dataset
The persona-vectors paper draws its Sycophancy arm of the Trait-eliciting finetuning datasets (Evil, Sycophancy, Hallucination) directly from this source: roughly 10,000 questions (100 held out per category for later evaluation) are paired with Claude 3.7 Sonnet-generated responses at three levels — an honest and independent Normal response, a polite but sycophantic "I" response, and a more extreme, obviously flattering "II" response (persona-vectors, §"D.1 Question collection.", p. 35; §"D.2 Response generation", p. 36). Unlike the Evil (persona trait) and Hallucination (closed-domain fabrication) datasets, which are generated wholesale by an LLM because no adequate existing dataset covered them, sycophancy's questions are drawn from this pre-existing benchmark rather than synthesized from scratch.
External validation of the sycophancy trait
The 300 questions withheld from the training split (100 per category) are reused separately as an out-of-distribution check on the sycophancy persona vector: the paper's own sycophancy evaluation prompt is applied to a single rollout per held-out question, and the resulting trait expression scores correlate strongly with scores on the paper's own 20-question evaluation set — r = 0.964 on Qwen and r = 0.952 on Llama (persona-vectors, §"B.3 Additional evaluations on standard benchmarks", p. 29). Unlike HaluEval's check on hallucination, which draws on a benchmark built by an entirely separate research effort, this is a held-out split of the very same source used to build the training data — a weaker but still informative test, since it rules out overfitting to the paper's own 20 extraction and evaluation questions specifically, even if not to the dataset's underlying opinion-agreement format.