Coherence score
Persona Vectors: Monitoring and Controlling Character Traits in Language Models — inherited
A 0-100 quality metric, following Betley et al. (2025)'s methodology, in which GPT-4.1-mini rates a response's general coherence. Used throughout the paper's steering experiments as a check that trait-suppressing or trait-inducing interventions have not degraded the model into incoherent output (e.g. requiring average coherence above 75-80 across reported results).
An intervention that pushes a trait score toward zero could be doing that honestly, or it could just be breaking the model into gibberish that no longer expresses much of anything — and those two outcomes look identical on the trait metric alone. This borrowed quality check exists to tell them apart.
A judge model rates each response's general coherence, independent of any trait — that mechanism opens the account, alongside the two different pass thresholds applied depending on which mitigation is being evaluated. Pairing this check with a separate general-capability metric matters too, since a response can stay perfectly coherent while still losing real ability along the way.
A borrowed quality check
Coherence score is not this paper's invention: it follows the scoring methodology of Betley et al. (2025), under which each response is rated 0-100 by GPT-4.1-mini for its general coherence, independent of any specific trait (persona-vectors, §"5.1 Post-hoc steering mitigates behavioral shifts", p. 8). The paper needs this check because steering interventions can degrade a model into incoherent text rather than genuinely inducing or suppressing a trait — a known failure mode it cites directly from Durmus et al. (2024b) — and a sufficiently broken response could register an extreme trait expression score for the wrong reason, since garbled text can accidentally look like it lacks (or exhibits) the trait being measured. Coherence score and trait expression score are graded by the same judge model but on opposite kinds of rubric: trait expression is scored against a rubric manufactured fresh per trait by Trait-specific artifact generation, while coherence uses one fixed rubric unchanged across every trait and every experiment.
Two different floors: 75 versus 80
The paper enforces coherence as a gate rather than merely reporting it, and it sets that gate at two different levels depending on which intervention is being evaluated. Results for inference-time steering and post-hoc steering are only reported once "average response coherence is above 75" (persona-vectors, §"5.1 Post-hoc steering mitigates behavioral shifts", p. 8). Results for preventative steering, the paper's preferred and more novel method, are held to a stricter floor: an "average coherence score across all models above 80" (persona-vectors, §"5.2 Preventative steering limits behavioral shifts during finetuning", p. 8). The stricter bar on preventative steering matches its role as the method the paper is arguing for — a trait-expression-score result is treated as a genuine behavioral effect only once its coherence has cleared the relevant threshold, functioning as a validity check on the rest of the paper's steering results rather than a headline finding in its own right.
Paired with a capability check
Coherence score is not the paper's only quality safeguard on steering results — it is paired with a general capability check as well: MMLU accuracy, tracked as a gray line alongside every steering-coefficient sweep the paper reports (persona-vectors, §"5.1 Post-hoc steering mitigates behavioral shifts", p. 8). The two checks catch different failure modes: coherence catches a response that has become gibberish at the sentence level, while MMLU catches a broader loss of general capability that could otherwise survive as fluent but degraded reasoning. Inference-time (post-hoc) steering shows MMLU accuracy dropping at large steering coefficients even while coherence stays above its floor, whereas preventative steering "better preserves the model's general capabilities" on the same metric (persona-vectors, §"5.2 Preventative steering limits behavioral shifts during finetuning", p. 8) — evidence that a passed coherence check does not by itself guarantee that capability has been preserved.