Transformer visualization via dictionary learning (Yun et al., 2021) — is extended by → Sparse autoencoders (SAEs)

Yun et al. (2021) train a single sparse dictionary across all of a language model's residual-stream layers simultaneously, treating each token's representation at every depth as a linear superposition of shared 'transformer factors.' This paper takes the opposite granularity: it trains a separate sparse autoencoder per layer, and separately again for MLP sublayers in Appendix C.3, which is what makes it meaningful to compare interpretability and dead-feature rates layer by layer (Figures 2, 9, 16) rather than only at the level of the whole residual stream. The larger difference is evaluative rather than architectural -- Yun et al. never test whether their learned factors are more interpretable than baselines, more monosemantic than neurons, or more causally precise than PCA on a behavioral task, which is exactly the autointerpretability scoring, case-study, and IOI-patching apparatus this paper adds on top of the same basic sparse-dictionary-learning idea.