MMLU (Massive Multitask Language Understanding)
Steering Llama 2 via Contrastive Activation Addition — inherited; Persona Vectors: Monitoring and Controlling Character Traits in Language Models — refined
A benchmark of multiple-choice questions across 57 subjects (Hendrycks et al., 2021) used, unmodified, to test whether CAA steering causes adverse side effects on a model's general knowledge and problem-solving capabilities.
Pushing a model's activations around to change one behavior risks breaking something else you weren't trying to touch, general knowledge or reasoning, the basic competence a model needs for everything unrelated to the behavior being steered. Checking for that kind of collateral damage needs a benchmark that has nothing to do with any of the seven target behaviors.
MMLU plays exactly that role here, a broad multiple-choice test spanning 57 subjects, used unmodified to confirm that steering the model toward or away from a target behavior leaves its general problem-solving performance largely intact.
57 subjects as a general-knowledge probe
MMLU (Massive Multitask Language Understanding) is a benchmark of multiple-choice questions spanning 57 subjects — from elementary mathematics and US history to moral scenarios, professional law, and clinical knowledge — assembled by Hendrycks et al. (2021) to test the breadth of a language model's general knowledge and reasoning rather than any single narrow skill. It predates CAA by two years and is inherited here unmodified as an off-the-shelf check rather than a target of study: the paper randomly samples ten questions from each of the 57 categories (570 total), reformats them as two-option A/B multiple-choice, and reports the average probability the model assigns to the correct answer — the same scoring convention Multiple-choice behavioral evaluation and layer sweep uses for its own behavior scores (contrastive-activation-addition, §"7 Effect of CAA on general capabilities", p. 6).
The capability floor CAA is checked against
MMLU's role here is purely defensive: it is the test that nothing else broke while a behavior was being steered. With no intervention, Llama 2 13B Chat averages 0.63 probability on the correct answer; adding the steering vector at layer 14 with multiplier +1 or −1 shifts that average by at most 0.06 for every one of the seven behaviors — Corrigibility moves to 0.64/0.59, Hallucination to 0.64/0.57 (contrastive-activation-addition, §"7 Effect of CAA on general capabilities", p. 6). That is a far narrower swing than the same multipliers produce on the matching Multiple-choice behavioral evaluation and layer sweep score, evidence CAA is finding something closer to a specific behavioral direction than a blunt disruption of the network. The result reruns, with a much smaller measured cost, the question InstructGPT's Alignment tax first named for RLHF finetuning, and complements SuperGLUE's narrower linguistic-inference check with a broader probe of stored knowledge across independent domains.
2025: MMLU as the capability gate every mitigation is measured against
Persona Vectors inherits MMLU unmodified from CAA's own capability check and reuses it as the recurring floor every intervention in the paper is measured against, rather than introducing a new capability benchmark of its own. Post-hoc, inference-time steering against a persona vector after finetuning reduces trait expression, but at large steering coefficients it visibly degrades MMLU accuracy — "similar to findings in Durmus et al. (2024b)" — establishing MMLU as the cost side of the steering-versus-capability tradeoff the rest of the paper's mitigations are judged against (persona-vectors, §"5.1 Post-hoc steering mitigates behavioral shifts", p. 7). Preventative steering, which adds the vector during finetuning rather than after it, clears that same bar more cheaply: it "better preserves the model's general capabilities compared to inference-time steering, as measured by MMLU accuracy," and extending it across multiple layers via Multi-layer steering with layer-incremental vectors pushes trait expression to near-baseline on even the hardest datasets "without incurring any MMLU degradation compared to regular finetuning" (persona-vectors, §"5.2 Preventative steering limits behavioral shifts during finetuning", p. 8; p. 9). MMLU also gates whether preventative steering is safe to apply by default: run on a benign, non-trait-inducing dataset, it has "only a negligible effect on MMLU accuracy," evidence the intervention costs nothing when there is no drift to cancel (persona-vectors, §"J.6 Assessing side effects of preventative steering on benign data", p. 48). The clearest single contrast comes from the fact-acquisition case study: teaching a model 1,000 new facts and then suppressing the resulting hallucination spike, inference-time steering "tends to break the model," substantially degrading MMLU, while preventative steering "only slightly reduces new-fact accuracy while largely preserving MMLU performance" (persona-vectors, §"J.7 Case study: a fact-acquisition task", p. 48; p. 49). Across every one of these experiments, MMLU functions less as a headline result than as the fixed instrument that makes claims like "preventative steering preserves capability better than post-hoc steering" falsifiable at all.