SAE decomposition of persona vectors — applies the unsupervised machinery of → Sparse autoencoders (SAEs)

Applying sparse autoencoders here means training BatchTopK SAEs (Bussmann et al., 2024) at every fourth layer of Qwen-2.5-7B-Chat's residual stream, using a dictionary-learning codebase adapted from Marks et al. (2024) (persona-vectors, §"M.1 SAE training details", p. 59). Each SAE has 131,072 features against a residual-stream dimension of 3,584 -- roughly a 37x expansion factor -- and the paper reports results at sparsity level k=64 active features per token. This is a substantially heavier interpretability investment than anything else in the paper: where the persona-vector extraction pipeline needs only 400 rollouts per trait (20 extraction questions, times 10 rollouts, times two contrastive system prompts) and a single mean-difference computation to produce a usable direction (§"2.2 Extracting persona vectors", p. 3), training even one BatchTopK SAE at this width requires the standard large-scale unsupervised dictionary-learning procedure, run separately for each of several residual-stream layers. The payoff is a fixed, trait-agnostic decomposition of the whole residual stream computed once, into which any persona vector -- evil, sycophancy, hallucination, or a trait never explicitly targeted by the extraction pipeline -- can later be projected and decomposed, in contrast to the persona-vector method's need to rerun contrastive extraction separately for every new trait.