Fact-acquisition case study
Persona Vectors: Monitoring and Controlling Character Traits in Language Models — introduced
A case study finetuning models on 1,000 facts postdating the training cutoff, showing that learning new facts substantially increases hallucination. Used to test whether preventative and post-hoc steering can suppress this induced hallucination without erasing the newly learned facts or degrading MMLU; preventative steering succeeds with only a modest reduction in new-fact accuracy, while inference-time steering tends to break the model.
Suppressing an unwanted trait sounds straightforwardly good until the trait happens to be entangled with something worth keeping — in this case, brand-new factual knowledge a model was deliberately finetuned to learn, which measurably increases its tendency to hallucinate along the way. This case study is built to test whether a fix for one problem quietly destroys the other.
Constructing the dataset of postdating facts comes first, before the account turns to what happens when the induced hallucination is corrected after training versus during it — one approach breaks the model outright, the other barely touches the new knowledge. A related comparison against a purely prompt-based version of the same protective idea rounds things out.
The setup
The case study is built to test a specific worry: could suppressing hallucination via steering also suppress genuinely new, useful knowledge a finetuning run is meant to teach? The paper constructs "a dataset of 1,000 new facts that occurred after the model's training cut-off date," formatted as chat conversations incorporating these facts (persona-vectors, §"J.7.1 Preventative steering on a fact-acquisition task", p. 48). Finetuning on this dataset produced "a substantial increase in the model's tendency to hallucinate" relative to the base model's hallucination score of roughly 20, a self-contained replication, on a chat model, of a pattern Gekhman et al. (2024) had previously shown only for base models: that learning new facts through finetuning increases hallucination rate (persona-vectors, §"7 Related work", p. 12; §"J.7 Case study: a fact-acquisition task", p. 48).
What the steering comparison shows
Both inference-time and preventative steering reduce the induced hallucination back to baseline, but with very different collateral cost. Inference-time (post-hoc) steering "tends to break the model" outright on this task, substantially degrading both MMLU and new-fact accuracy along the way. Preventative steering, at a coefficient of 1.25, brings hallucination fully back to baseline while reducing new-fact accuracy only slightly and leaving MMLU "essentially unchanged" (persona-vectors, §"J.7.1 Preventative steering on a fact-acquisition task", p. 48-49). Because genuinely new factual content plausibly sits near a hallucination-adjacent direction in activation space -- structurally, a model stating claims it cannot independently verify looks like hallucination whether or not the claims happen to be true -- this is a real risk the case study confirms does not materialize: the two effects turn out separable enough that steering hard against hallucination does not require sacrificing the newly learned facts.
The prompting comparison folded into the same study
The same case study also hosts the direct comparison between preventative steering and Preventative prompting: at a moderate coefficient (0.5), steering performs comparably to prepending a hallucination-eliciting system prompt to every training example, across hallucination score, new-fact accuracy, and MMLU alike, while a higher steering coefficient (1.25) reaches a level of suppression prompting has no equivalent way to match (persona-vectors, §"J.7.2 Preventative prompting on a fact-acquisition task", p. 49). Because the fact-acquisition setting is the paper's only case study built around a genuine, desirable side effect a mitigation could plausibly destroy -- new knowledge, not just an unwanted trait -- it is also the setting the paper leans on to earn the "vaccine" framing of preventative steering empirically, rather than asserting it by definition.