HaluEval
Persona Vectors: Monitoring and Controlling Character Traits in Language Models — inherited
A prior hallucination-detection benchmark (Li et al., 2023) whose QA split's first 1,000 questions are used as an external, out-of-distribution question set to confirm that hallucination trait expression scores measured on the paper's own evaluation questions generalize to a standard benchmark.
A measurement that only ever gets checked against the same handful of questions used to build it hasn't really been tested — it just agrees with itself. Real confidence in a metric requires checking it against something built entirely independently, by people with no stake in the result matching.
The benchmark's contents open the account, along with how a slice of its questions gets repurposed here purely as an outside check on the paper's own hallucination measurements. Why this particular check counts as the strongest of the paper's three external validations comes last — built by a separate research effort, for a different purpose, with no awareness the paper's evaluation rubric would ever exist.
What HaluEval is
HaluEval is a prior hallucination-detection benchmark (Li et al., 2023), built independently of this paper to evaluate how well large language models can recognize when a piece of text — their own or someone else's — contains fabricated or unsupported content. It combines human-annotated and machine-generated hallucinated samples across question-answering, knowledge-grounded dialogue, and text-summarization tasks, pairing each with a corresponding non-hallucinated reference so that a model's or judge's discrimination ability can be scored directly. The persona-vectors paper uses only the QA split, and only as an external test set: it plays no role in constructing any of the paper's own training data.
Use in the paper: external validation for the hallucination trait
The paper takes the first 1,000 questions from HaluEval's QA split and applies its own Hallucination (closed-domain fabrication) evaluation prompt to the finetuned models' responses, exactly as it does for its own 20-question evaluation set (persona-vectors, §"B.3 Additional evaluations on standard benchmarks", p. 29). The resulting trait expression scores correlate strongly with scores on the paper's own evaluation questions: r = 0.855 on Qwen and r = 0.942 on Llama (persona-vectors, §"B.3 Additional evaluations on standard benchmarks", p. 29). This checks whether the hallucination persona vector, and the Trait expression score (LLM judge) judge built to measure it, are tracking a real behavioral tendency or merely overfit to the paper's own hand-picked 20 questions.
The strongest of the paper's three external checks
Among the paper's three main traits, hallucination's external check is the most independent of the three. Sycophancy's check reuses a held-out split of Sycophancy dataset (Nishimura-Gasparian et al., 2024), the very same source used to build its training data, and evil's check reruns Betley et al.'s own evil-scoring protocol on the paper's own models. HaluEval, by contrast, was built entirely separately, by a different research group, for a different purpose — hallucination detection generally, not persona-vector research — before this paper existed. Because its questions were authored with no awareness of the persona-vectors project's evaluation rubric or extraction pipeline, it rules out a stronger form of overfitting than the other two checks can: not just overfitting to twenty specific questions, but overfitting to the whole style and construction process behind them.